added everything

README.md

@@ -1,141 +1,229 @@
----
-license: gpl-3.0
-title: Kudasai
-sdk: gradio
-emoji: 🈷️
-python_version: 3.10.0
-app_file: webgui.py
-colorFrom: gray
-colorTo: gray
-short_description: Japanese-English preprocessor with automated translation.
-pinned: true
----
-
 ---------------------------------------------------------------------------------------------------------------------------------------------------
 **Table of Contents**
 
-- [Notes](#notes)
-- [Naming Conventions](#naming-conventions)
-- [General Usage](#general-usage)
-- [Indexing and Preprocessing](#kairyou)
-- [Translation with DeepL](#kaiseki)
-- [Translation with LLMs](#kijiku)
-- [Translation with LLMs Settings](#translation-with-llms-settings)
-- [Web GUI](#webgui)
-- [License](#license)
-- [Contact](#contact)
+- [**Notes**](#notes)
+- [**Dependencies**](#dependencies)
+- [**Quick Start**](#quick-start)
+- [**Command Line Interface (CLI)**](#command-line-interface-cli)
+  - [Usage](#usage)
+    - [Preprocess Mode](#preprocess-mode)
+    - [Translate Mode](#translate-mode)
+  - [Additional Notes](#additional-notes)
+- [**Preprocessing**](#preprocessing)
+- [**Translator**](#translator)
+- [**Translator Settings**](#translator-settings)
+- [**Web GUI**](#web-gui)
+- [**Hugging Face**](#hugging-face)
+- [**License**](#license)
+- [**Contact**](#contact)
+- [**Acknowledgements**](#acknowledgements)
 
 ---------------------------------------------------------------------------------------------------------------------------------------------------
-**Notes**<a name="notes"></a>
+## **Notes**<a name="notes"></a>
+
+Windows 10 and Linux Mint are the only tested operating systems, feel free to test on other operating systems and report back to me. I will do my best to fix any issues that arise.
 
-This readme is for the Hugging Space instance of Kudasai's WebGUI and the WebGUI itself, to run Kudasai locally or see any info on the project, please see the [GitHub Page](https://github.com/Bikatr7/Kudasai).
+To see the README for the Hugging Face hosted version of Kudasai, please see [here](https://github.com/Bikatr7/Kudasai/blob/main/lib/gui/HUGGING_FACE_README.md). Further WebGUI documentation can be found there as well.
+
+Python version: 3.10+
 
 Streamlining Japanese-English Translation with Advanced Preprocessing and Integrated Translation Technologies.
 
 Preprocessor and Translation logic is sourced from external packages, which I also designed, see [Kairyou](https://github.com/Bikatr7/Kairyou) and [EasyTL](https://github.com/Bikatr7/easytl) for more information.
 
-Kudasai has a public trello board, you can find it [here](https://trello.com/b/Wsuwr24S/kudasai) to see what I'm working on.
+Kudasai has a public trello board, you can find it [here](https://trello.com/b/Wsuwr24S/kudasai) to see what I'm working on and what's coming up.
 
-The webgui on huggingface does not save anything through runs, so you will need to download the output files or copy the text out of the webgui. API keys are not saved, and the output folder is overwritten every time you run it. Archives deleted every run as well.
+Kudasai is proud to have been a Backdrop Build v3 Finalist:
+https://backdropbuild.com/builds/v3/kudasai
 
 ---------------------------------------------------------------------------------------------------------------------------------------------------
-**Naming Conventions**<a name="naming-conventions"></a> 
+## **Dependencies**<a name="dependencies"></a>
 
-kudasai.py - Main script - ください　- Please
+backoff==2.2.1
 
-Kairyou - Preprocessing Package - 改良 - Reform
+gradio==4.20.0
 
-kaiseki.py - DeepL translation module - 解析 - Parsing
+kairyou==1.5.0
 
-kijiku.py - OpenAI translation module - 基軸 - Foundation
+easytl==0.3.3
 
-Kudasai gets it's original name idea from it's inspiration, Atreyagaurav's Onegai. Which also means please. You can find that [here](https://github.com/Atreyagaurav/onegai)
+or see requirements.txt
+
+Also requires spacy's ja_core_news_lg model, which can be installed via the following command:
+
+```bash
+python -m spacy download ja_core_news_lg
+```
+
+or on Linux
+
+```bash
+python3 -m spacy download ja_core_news_lg
+```
 
 ---------------------------------------------------------------------------------------------------------------------------------------------------
+## **Quick Start**<a name="quick-start"></a>
 
-**General Usage**<a name="general-usage"></a>
+Windows is assumed for the rest of this README, but the process should be similar for Linux. This is for the console version, for something less linear, see the [Web GUI](#webgui) section.
 
-Kudasai's WebGUI is pretty easy to understand for the general usage, most incorrect actions will be caught by the system and a message will be displayed to the user on how to correct it.
+Due to PyPi limitations, you need to install SpaCy's JP Model, which can not be included automatically due to it being a direct dependency link which PyPi does not support. Make sure you do this after installing the requirements.txt file as it requires Kairyou/SpaCy to be installed first.
 
-Normally, Kudasai would save files to the local system, but on Hugging Face's servers, this is not possible. Instead, you'll have to click the 'Save As' button to download the files to your local system.
+```bash
+python -m spacy download ja_core_news_lg
+```
 
-Or you can click the copy button on the top right of textbox modals to copy the text to your clipboard.
+Simply run Kudasai.py, enter a txt file path to the text you wish to preprocess/translate, and then insert a replacement json file path if you wish to use one. If you do not wish to use a replacement json file, you can simply input a blank space and Kudasai will skip preprocessing and go straight to translation.
 
-For further details, see below chapters.
+Kudasai will offer to index the text, which is useful for finding new names to add to the replacement json file. This is optional and can be skipped.
 
----------------------------------------------------------------------------------------------------------------------------------------------------
+After preprocessing is completed (if triggered), you will be prompted to choose a translation method.
+
+You can choose between OpenAI, Gemini, and DeepL. Each have their own pros and cons, but OpenAI is the recommended translation method. DeepL and Gemini currently offer free versions, but all three require an api key, you will be prompted to enter this key when you choose to run the translation module.
+
+Next, Kudasai will ask you to confirm it's settings. This can be overwhelming, but you can simply enter 1 to confirm and use the default settings. If you wish to change them, you can do so here.
+
+See the [**Translator Settings**](#translator-settings) section for more information on Kudasai's Translation settings, but default should run fine. Inside the demo folder is a copy of the settings I use to translate COTE should you wish to use them. There is also a demo txt file in the demo folder that you can use to test Kudasai.
+
+Kudasai will then ask if you want to change your api key, simply enter 2 for now. 
+
+Next Kudasai will display an estimated cost of translation, this is based on the number of tokens in the preprocessed text as determined by tiktoken for OpenAI, by Google for Gemini, and by DeepL for DeepL. Kudasai will then prompt for confirmation, if this is fine, enter 1 to run the translation module otherwise 2 to exit.
+
+Kudasai will then run the translation module and output the translated text and other logs to the output folder in the same directory as Kudasai.py.
 
-**Indexing and Preprocessing**<a name="kairyou"></a>
+These files are:
 
-Indexing is not for everyone, only use it if you have a large amount of previous text and want to flag new names. It can be a very slow and long process, especially on Hugging Face's servers. It's recommended to use a local version of Kudasai for this process.
+    "debug_log.txt" : A log of crucial information that occurred during Kudasai's run, useful for debugging or reporting issues as well as seeing what was done.
 
-You'll need a txt file or some text to index. You'll also need a knowledge base, this can either be a single txt file or a directory of them, as well as a replacements json. Either Kudasai or Fukuin Type works. See [this](https://github.com/Bikatr7/Kairyou?tab=readme-ov-file#kairyou) for further details on replacement jsons.
+    "error_log.txt" : A log of errors that occurred during Kudasai's run if any, useful for debugging or reporting issues.
 
-Please do indexing before preprocessing, output is neater that way.
+    "je_check_text.txt" : A log of the Japanese and English sentences that were paired together, useful for checking the accuracy of the translation and further editing of a machine translation.
 
-For Preprocessing, you'll need a txt file or some text to preprocess. You'll also need a replacements json. Either Kudasai or Fukuin Type works like with indexing.
+    "preprocessed_text.txt" : The preprocessed text, the text output by Kairyou (preprocessor).
 
-For both, text is put in the textbox modals, with the output text being in the first field, and results being in the second field. 
+    "preprocessing_results.txt" : A log of the results of the preprocessing, shows what was replaced and how many times.
 
-They both have a debug field, but neither module really uses it.
+    "translated_text.txt" : The translated text, the text output by Kaiseki or Kijiku.
+
+Old runs are stored in the archive folder in output as well.
+
+If you have any questions, comments, or concerns, please feel free to open an issue.
 
 ---------------------------------------------------------------------------------------------------------------------------------------------------
+## **Command Line Interface (CLI)**<a name="cli"></a>
+
+Kudasai provides a Command Line Interface (CLI) for preprocessing and translating text files. This section details how to use the CLI, including the required and optional arguments for each mode.
+
+### Usage
+
+The CLI supports two modes: `preprocess` and `translate`. Each mode requires specific arguments to function properly.
+
+#### Preprocess Mode
+
+The `preprocess` mode preprocesses the text file using the provided replacement JSON file. 
+
+**Command Structure:**
+
+```bash
+python path_to_kudasai.py preprocess <input_file> <replacement_json> [<knowledge_base>]
+```
+
+**Required Arguments:**
+- `<input_file>`: Path to the text file to preprocess.
+- `<replacement_json>`: Path to the replacement JSON file.
+
+**Optional Arguments:**
+- `<knowledge_base>`: Path to the knowledge base file (directory, file, or text).
+
+**Example:**
+
+```bash
+python C:\\path\\to\\kudasai.py preprocess "C:\\path\\to\\input_file.txt" "C:\\path\\to\\replacement_json.json" "C:\\path\\to\\knowledge_base"
+```
+
+#### Translate Mode
 
-**Translation with DeepL**<a name="kaiseki"></a>
+The `translate` mode translates the text file using the specified translation method.
 
-DeepL is a paid service, so you'll need an API key to use it. You can get one [here](https://www.deepl.com/pro-api). However is free under 500,000 characters a month.
+**Command Structure:**
 
-Same general things apply here, use a text input (File or raw text). Also enter your API key in the API key field. 
+```bash
+python path_to_kudasai.py translate <input_file> <translation_method> [<translation_settings_json>] [<api_key>]
+```
 
-While DeepL translation does work, it is currently deprecated in favor of the LLMs, and a bit buggy. It's recommended to use the LLMs for translation. Perhaps in the future, I'll update the DeepL translation to be more stable given demand.
+**Required Arguments:**
+- `<input_file>`: Path to the text file to translate.
 
-DeepL translation is fairly unsophisticated compared to the LLMs, so there's not much to configure. Press the translate button and wait for the results. Output will show in the appropriate fields.
+**Optional Arguments:**
+- `<translation_method>`: Translation method to use (`'deepl'`, `'openai'`, or `'gemini'`). Defaults to `'deepl'`.
+- `<translation_settings_json>`: Path to the translation settings JSON file (overrides current settings).
+- `<api_key>`: API key for the translation service. If not provided, it will use the in the settings directory or prompt for it if that's not found.
+
+**Example:**
+
+```bash
+python C:\\path\\to\\kudasai.py translate "C:\\path\\to\\input_file.txt" gemini "C:\\path\\to\\translation_settings.json" "YOUR_API_KEY"
+```
+
+### Additional Notes
+- All arguments should be enclosed in double quotes if they contain spaces. Double quotes are optional and will be stripped. Single quotes are not allowed.
 
 ---------------------------------------------------------------------------------------------------------------------------------------------------
 
-**Translation with LLMs**<a name="kijiku"></a>
+## **Preprocessing**<a name="preprocessing"></a>
+
+Preprocessing is the act of preparing text for translation by replacing certain words or phrases with their translated counterparts. 
 
-Kudasai supports 2 different LLMs at the moment, OpenAI's GPT and Google's Gemini. 
+Kudasai uses Kairyou for preprocessing, which is a powerful preprocessor that can replace text in a text file based on a json file. This is useful for replacing names, places, and other things that may not translate well or to simply speed up the translation process.
 
-For OpenAI, you'll need an API key, you can get one [here](https://platform.openai.com/docs/api-reference/authentication). This is a paid service with no free tier.
+You can run the preprocessor by using the CLI or simply running kudasai.py as instructed in the [Quick Start](#quick-start) section.
+
+Many replacement json files are included in the jsons folder, you can also make your own if you wish provided it follows the same format. See an example below
+Kudasai/Kairyou works with both Kudasai and Fukuin Json's, the below is a Kudasai type json.
+
+![Example JSON](https://i.imgur.com/u3FnUia.jpg)
+
+---------------------------------------------------------------------------------------------------------------------------------------------------
 
-For Gemini, you'll need an API key, you can get one [here](https://ai.google.dev/tutorials/setup). Gemini is free to use under 60 concurrent requests.
+## **Translator**<a name="translator"></a>
 
-I'd recommend using GPT for most things, as it's generally better at translation.
+Kudasai uses EasyTL for translation, which is a versatile translation library that uses several translation APIs to translate text.
 
-Once again, mostly straightforward, fill in your API key, select your LLM, and select your text. You'll also need to add your settings file if on HuggingFace.
+Kudasai currently supports OpenAI, Gemini, and DeepL for translation. OpenAI is the recommended translation method, but DeepL and Gemini are also good alternatives.
 
-You can calculate costs here or just translate. Output will show in the appropriate fields.
+You can run the translator by running kudasai.py as instructed in the [Quick Start](#quick-start) section.
 
-For further details on the settings file, see [here](#translation-with-llms-settings).
+Note that you need an API key for OpenAI, Gemini, and DeepL. You will be prompted to enter this key when you choose to run the translation module.
+
+The translator has a lot of settings, simply using the default settings is fine or the one provided in the demo folder. You can also change these manually when confirming your settings, as well as loading a custom json as your settings by pressing c at this window, with the settings in the script directory.
+
+The settings are fairly complex, see the below section [Translator Settings](#translator-settings) for more information.
 
 ---------------------------------------------------------------------------------------------------------------------------------------------------
 
-**Translation with LLMs Settings**<a name="translation-with-llms-settings"></a>
+## **Translator Settings**<a name="translator-settings"></a>
 
 (Fairly technical, can be abstracted away by using default settings or someone else's settings file.)
 
-    ----------------------------------------------------------------------------------
-    Kijiku Settings:
+    Base Translation Settings:
 
-    prompt_assembly_mode : 1 or 2. 1 means the system message will actually be treated as a system message. 2 means it'll be treated as a user message. 1 is recommend for gpt-4 otherwise either works. For Gemini, this setting is ignored.
+    prompt_assembly_mode : 1 or 2. 1 means the system message will actually be treated as a system message. 2 means it'll be treated as a user message. 1 is recommend for gpt-4 otherwise either works. For Gemini & DeepL, this setting is ignored.
 
-    number_of_lines_per_batch : The number of lines to be built into a prompt at once. Theoretically, more lines would be more cost effective, but other complications may occur with higher lines. So far been tested up to 48.
+    number_of_lines_per_batch : The number of lines to be built into a prompt at once. Theoretically, more lines would be more cost effective, but other complications may occur with higher lines. So far been tested up to 48 by me.
 
-    sentence_fragmenter_mode : 1 or 2  (1 - via regex and other nonsense) 2 - None (Takes formatting and text directly from API return)) the API can sometimes return a result on a single line, so this determines the way Kijiku fragments the sentences if at all. Use 2 for newer models.
+    sentence_fragmenter_mode : 1 or 2  (1 - via regex and other nonsense) 2 - None (Takes formatting and text directly from API return)) the API can sometimes return a result on a single line, so this determines the way Kudasai fragments the sentences if at all. Use 2 for newer models and Deepl.
 
-    je_check_mode : 1 or 2, 1 will print out the jap then the english below separated by ---, 2 will attempt to pair the english and jap sentences, placing the jap above the eng. If it cannot, it will default to 1. Use 2 for newer models.
+    je_check_mode : 1 or 2, 1 will print out the jap then the english below separated by ---, 2 will attempt to pair the english and jap sentences, placing the jap above the eng. If it cannot, it will default to 1. Use 2 for newer models and DeepL.
 
-    number_of_malformed_batch_retries : (Malformed batch is when je-fixing fails) How many times Kijiku will attempt to mend a malformed batch (mending is resending the request), only for gpt4. Be careful with increasing as cost increases at (cost * length * n) at worst case. This setting is ignored if je_check_mode is set to 1.
+    number_of_malformed_batch_retries : (Malformed batch is when je-fixing fails) How many times Kudasai will attempt to mend a malformed batch (mending is resending the request). Be careful with increasing as cost increases at (cost * length * n) at worst case. This setting is ignored if je_check_mode is set to 1.
 
-    batch_retry_timeout : How long Kijiku will try to translate a batch in seconds, if a requests exceeds this duration, Kijiku will leave it untranslated.
+    batch_retry_timeout : How long Kudasai will try to translate a batch in seconds, if a requests exceeds this duration, Kudasai will leave it untranslated.
 
-    number_of_concurrent_batches : How many translations batches Kijiku will send to the translation API at a time. For OpenAI, be conservative as rate-limiting is aggressive, I'd suggest 3-5. For Gemini, do not exceed 60.
+    number_of_concurrent_batches : How many translations batches Kudasai will send to the translation API at a time. For OpenAI, be conservative as rate-limiting is aggressive, I'd suggest 3-5. For Gemini, do not exceed 15 for 1.0 or 2 for 1.5. This setting more or less doesn't matter for DeepL.
     ----------------------------------------------------------------------------------
     Open AI Settings:
     See https://platform.openai.com/docs/api-reference/chat/create for further details
     ----------------------------------------------------------------------------------
-    openai_model : ID of the model to use. Kijiku only works with 'chat' models.
+    openai_model : ID of the model to use. Kudasai only works with 'chat' models.
 
     openai_system_message : Instructions to the model. Basically tells the model how to translate.
 
@@ -143,13 +231,13 @@ For further details on the settings file, see [here](#translation-with-llms-sett
 
     openai_top_p : An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered. I generally recommend altering this or temperature but not both.
 
-    openai_n : How many chat completion choices to generate for each input message. Do not change this.
+    openai_n : How many chat completion choices to generate for each input message. Do not change this, as Kudasai will always use 1.
 
-    openai_stream : If set, partial message deltas will be sent, like in ChatGPT. Tokens will be sent as data-only server-sent events as they become available, with the stream terminated by a data: [DONE] message. See the OpenAI python library on GitHub for example code. Do not change this.
+    openai_stream : If set, partial message deltas will be sent, like in ChatGPT. Tokens will be sent as data-only server-sent events as they become available, with the stream terminated by a data: [DONE] message. See the OpenAI python library on GitHub for example code. Do not change this as Kudasai does not support this feature.
 
-    openai_stop : Up to 4 sequences where the API will stop generating further tokens. Do not change this.
+    openai_stop : Up to 4 sequences where the API will stop generating further tokens. Do not change this as Kudasai does not support this feature.
 
-    openai_logit_bias : Modifies the likelihood of specified tokens appearing in the completion. Do not change this.
+    openai_logit_bias : Modifies the likelihood of specified tokens appearing in the completion. Do not change this as Kudasai does not support this feature.
 
     openai_max_tokens :  The maximum number of tokens to generate in the chat completion. The total length of input tokens and generated tokens is limited by the model's context length. I wouldn't recommend changing this. Is none by default. If you change to an integer, make sure it doesn't exceed that model's context length or your request will fail and repeat till timeout.
 
@@ -157,12 +245,12 @@ For further details on the settings file, see [here](#translation-with-llms-sett
 
     openai_frequency_penalty : Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model's likelihood to repeat the same line verbatim. Negative values encourage repetition. Should leave this at 0.0.
     ----------------------------------------------------------------------------------
-    openai_stream, openai_logit_bias, openai_stop and openai_n are included for completion's sake, current versions of Kudasai will hardcode their values when validating the Kijiku_rule.json to their default values. As different values for these settings do not have a use case in Kudasai's current implementation.
+    openai_stream, openai_logit_bias, openai_stop and openai_n are included for completion's sake, current versions of Kudasai will hardcode their values when validating the translation_settings.json to their default values. As different values for these settings do not have a use case in Kudasai's current implementation.
     ----------------------------------------------------------------------------------
     Gemini Settings:
-    https://ai.google.dev/docs/concepts#model-parameters for further details
+    See https://ai.google.dev/docs/concepts#model-parameters for further details
     ----------------------------------------------------------------------------------
-    gemini_model : The model to use. Currently only supports gemini-pro and gemini-pro-vision, the 1.0 model and it's aliases.
+    gemini_model : The model to use. Currently only supports gemini-pro and gemini-pro-vision, the 1.0 model and 1.5 models and their aliases.
 
     gemini_prompt : Instructions to the model. Basically tells the model how to translate.
 
@@ -172,43 +260,69 @@ For further details on the settings file, see [here](#translation-with-llms-sett
 
     gemini_top_k : Determines the number of most probable tokens to consider for each selection step. A higher value increases diversity, a lower value makes the output more deterministic.
 
-    gemini_candidate_count : The number of candidates to generate for each input message. Do not change this.
+    gemini_candidate_count : The number of candidates to generate for each input message. Do not change this as Kudasai will always use 1.
 
-    gemini_stream : If set, partial message deltas will be sent, like in Gemini Chat. Tokens will be sent as data-only server-sent events as they become available, with the stream terminated by a data: [DONE] message. Do not change this.
+    gemini_stream : If set, partial message deltas will be sent, like in Gemini Chat. Tokens will be sent as data-only server-sent events as they become available, with the stream terminated by a data: [DONE] message. Do not change this as Kudasai does not support this feature.
 
-    gemini_stop_sequences : Up to 4 sequences where the API will stop generating further tokens. Do not change this.
+    gemini_stop_sequences : Up to 4 sequences where the API will stop generating further tokens. Do not change this as Kudasai does not support this feature.
 
     gemini_max_output_tokens : The maximum number of tokens to generate in the chat completion. The total length of input tokens and generated tokens is limited by the model's context length. I wouldn't recommend changing this. Is none by default. If you change to an integer, make sure it doesn't exceed that model's context length or your request will fail and repeat till timeout.
     ----------------------------------------------------------------------------------
-    gemini_stream, gemini_stop_sequences and gemini_candidate_count are included for completion's sake, current versions of Kudasai will hardcode their values when validating the Kijiku_rule.json to their default values. As different values for these settings do not have a use case in Kudasai's current implementation.
+    gemini_stream, gemini_stop_sequences and gemini_candidate_count are included for completion's sake, current versions of Kudasai will hardcode their values when validating the translation_settings.json to their default values. As different values for these settings do not have a use case in Kudasai's current implementation.
+    ----------------------------------------------------------------------------------
+    Deepl Settings:
+    See https://developers.deepl.com/docs/api-reference/translate for further details
     ----------------------------------------------------------------------------------
+    deepl_context : The context in which the text should be translated. This is used to improve the translation. If you don't have any context, you can leave this empty. This is a DeepL Alpha feature and could be subject to change.
+
+    deepl_split_sentences : How the text should be split into sentences. Possible values are 'OFF', 'ALL', 'NO_NEWLINES'.
+
+    deepl_preserve_formatting : Whether the formatting of the text should be preserved. If you don't want to preserve the formatting, you can set this to False. Otherwise, set it to True.
+
+    deepl_formality : The formality of the text. Possible values are 'default', 'more', 'less', 'prefer_more', 'prefer_less'.
 
 ---------------------------------------------------------------------------------------------------------------------------------------------------
 
-**Web GUI**<a name="webgui"></a>
+## **Web GUI**<a name="webgui"></a>
 
-Below are images of the WebGUI:
+Kudasai also offers a Web GUI. It has all the main functionality of the program but in an easier and non-linear way.
 
-Indexing | Kairyou:
-![Indexing Screen | Kairyou](https://i.imgur.com/0a2mzOI.png)
+To run the Web GUI, simply run webgui.py which is in the same directory as kudasai.py
 
-Preprocessing | Kairyou:
-![Preprocessing Screen | Kairyou](https://i.imgur.com/2pt06gC.png)
+Below are some images of the Web GUI.
 
-Translation | Kaiseki:
-![Translation Screen | Kaiseki](https://i.imgur.com/X98JYsp.png)
+Detailed Documentation for this can be found on the Hugging Face hosted version of Kudasai [here](https://huggingface.co/spaces/Bikatr7/Kudasai/blob/main/README.md).
 
-Translation | Kijiku:
-![Translation Screen | Kijiku](https://i.imgur.com/X6IxyL8.png)
+Name Indexing | Kairyou:
+![Name Indexing Screen | Kairyou](https://i.imgur.com/QCPqjrw.jpeg)
 
-Kijiku Settings:
-![Kijiku Settings](https://i.imgur.com/VX0fGd5.png)
+Text Preprocessing | Kairyou:
+![Text Preprocessing Screen | Kairyou](https://i.imgur.com/r8nHEvw.jpeg)
 
-Logging:
-![Logging](https://i.imgur.com/IkUjpXR.png)
+Text Translation | Translator:
+![Text Translation Screen | Translator](https://i.imgur.com/0E9q2eh.jpeg)
+
+Translation Settings Page 1:
+![Translation Settings Page 1](https://i.imgur.com/0E9q2eh.jpeg)
+
+Translation Settings Page 2:
+![Translation Settings Page 2](https://i.imgur.com/8MQk6pL.jpeg)
+
+Logging Page:
+![Logging Page](https://i.imgur.com/vDPCUQC.jpeg)
+
+---------------------------------------------------------------------------------------------------------------------------------------------------
+
+## **Hugging Face**<a name="huggingface"></a>
+
+For those who are interested, or simply cannot run Kudasai locally, a instance of Kudasai's WebGUI is hosted on Hugging Face's servers. You can find it [here](https://huggingface.co/spaces/Bikatr7/Kudasai).
+
+It's a bit slower than running it locally, but it's a good alternative for those who cannot run it locally. The webgui on huggingface does not save anything through runs, so you will need to download the output files or copy the text out of the webgui. API keys are not saved, and the output folder is overwritten every time it loads. Archives deleted every run as well.
+
+To see the README for the Hugging Face hosted version of Kudasai, please see [here](https://huggingface.co/spaces/Bikatr7/Kudasai/blob/main/README.md).
 
 ---------------------------------------------------------------------------------------------------------------------------------------------------
-**License**<a name="license"></a>
+## **License**<a name="license"></a>
 
 This project (Kudasai) is licensed under the GNU General Public License (GPL). You can find the full text of the license in the [LICENSE](License.md) file.
 
@@ -217,12 +331,18 @@ The GPL is a copyleft license that promotes the principles of open-source softwa
 Please note that this information is a brief summary of the GPL. For a detailed understanding of your rights and obligations under this license, please refer to the full license text.
 
 ---------------------------------------------------------------------------------------------------------------------------------------------------
-**Contact**<a name="contact"></a>
+## **Contact**<a name="contact"></a>
 
-If you have any questions, comments, or concerns, please feel free to contact me at [Tetralon07@gmail.com](mailto:Tetralon07@gmail.com).
+If you have any questions, comments, or concerns, please feel free to contact me at [Bikatr7@proton.me](mailto:Bikatr7@proton.me)
 
 For any bugs or suggestions please use the issues tab [here](https://github.com/Bikatr7/Kudasai/issues).
 
-Once again, I actively encourage and welcome any feedback on this project.
+I actively encourage and welcome any feedback on this project.
+
+---------------------------------------------------------------------------------------------------------------------------------------------------
+
+## **Acknowledgements**<a name="acknowledgements"></a>
+
+Kudasai gets it's original name idea from it's inspiration, Atreyagaurav's Onegai. Which also means please. You can find that [here](https://github.com/Atreyagaurav/onegai)
 
 ---------------------------------------------------------------------------------------------------------------------------------------------------

demo/translation_settings.json

@@ -0,0 +1,45 @@
+{
+    "base translation settings": {
+        "prompt_assembly_mode": 1,
+        "number_of_lines_per_batch": 48,
+        "sentence_fragmenter_mode": 2,
+        "je_check_mode": 2,
+        "number_of_malformed_batch_retries": 1,
+        "batch_retry_timeout": 700,
+        "number_of_concurrent_batches": 2
+    },
+
+    "openai settings": {
+        "openai_model": "gpt-4-turbo",
+        "openai_system_message": "As a Japanese to English translator, translate narration into English simple past, everything else should remain in its original tense. Maintain original formatting, punctuation, and paragraph structure. Keep pre-translated terms and anticipate names not replaced. Preserve terms and markers marked with >>><<< and match the output's line count to the input's. Note: 〇 indicates chapter changes.",
+        "openai_temperature": 0.3,
+        "openai_top_p": 1.0,
+        "openai_n": 1,
+        "openai_stream": false,
+        "openai_stop": null,
+        "openai_logit_bias": null,
+        "openai_max_tokens": null,
+        "openai_presence_penalty": 0.0,
+        "openai_frequency_penalty": 0.0
+    },
+
+    "gemini settings": {
+        "gemini_model": "gemini-1.5-pro-latest",
+        "gemini_prompt": "As a Japanese to English translator, translate narration into English simple past, everything else should remain in its original tense. Maintain original formatting, punctuation, and paragraph structure. Keep pre-translated terms and anticipate names not replaced. Preserve terms and markers marked with >>><<< and match the output's line count to the input's. Note: 〇 indicates chapter changes.",
+        "gemini_temperature": 0.3,
+        "gemini_top_p": null,
+        "gemini_top_k": null,
+        "gemini_candidate_count": 1,
+        "gemini_stream": false,
+        "gemini_stop_sequences": null,
+        "gemini_max_output_tokens": null
+    },
+
+    "deepl settings":{
+        "deepl_context": "",
+        "deepl_split_sentences": "ALL",
+        "deepl_preserve_formatting": true,
+        "deepl_formality": "default"
+    }
+    
+}

handlers/json_handler.py

@@ -1,13 +1,13 @@
 ## built-in libraries
 import json
 import typing
+import logging
 
 ## third-party libraries
 from easytl import ALLOWED_GEMINI_MODELS, ALLOWED_OPENAI_MODELS
 
 ## custom modules
 from modules.common.file_ensurer import FileEnsurer
-from modules.common.logger import Logger
 from modules.common.toolkit import Toolkit
 
 ##-------------------start-of-JsonHandler---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
@@ -16,81 +16,15 @@ class JsonHandler:
 
     """
     
-    Handles the Kijiku Rules.json file and interactions with it.
+    Handles the translation_settings.json file and interactions with it.
 
     """
 
-    current_kijiku_rules = dict()
+    current_translation_settings = dict()
 
-    kijiku_settings_message = """
-----------------------------------------------------------------------------------
-Kijiku Settings:
+    with open(FileEnsurer.translation_settings_description_path, 'r', encoding='utf-8') as file:
+        translation_settings_message = file.read()
 
-prompt_assembly_mode : 1 or 2. 1 means the system message will actually be treated as a system message. 2 means it'll be treated as a user message. 1 is recommend for gpt-4 otherwise either works. For Gemini, this setting is ignored.
-
-number_of_lines_per_batch : The number of lines to be built into a prompt at once. Theoretically, more lines would be more cost effective, but other complications may occur with higher lines. So far been tested up to 48.
-
-sentence_fragmenter_mode : 1 or 2  (1 - via regex and other nonsense) 2 - None (Takes formatting and text directly from API return)) the API can sometimes return a result on a single line, so this determines the way Kijiku fragments the sentences if at all. Use 2 for newer models.
-
-je_check_mode : 1 or 2, 1 will print out the jap then the english below separated by ---, 2 will attempt to pair the english and jap sentences, placing the jap above the eng. If it cannot, it will default to 1. Use 2 for newer models.
-
-number_of_malformed_batch_retries : (Malformed batch is when je-fixing fails) How many times Kijiku will attempt to mend a malformed batch (mending is resending the request), only for gpt4. Be careful with increasing as cost increases at (cost * length * n) at worst case. This setting is ignored if je_check_mode is set to 1.
-
-batch_retry_timeout : How long Kijiku will try to translate a batch in seconds, if a requests exceeds this duration, Kijiku will leave it untranslated.
-
-number_of_concurrent_batches : How many translations batches Kijiku will send to the translation API at a time. For OpenAI, be conservative as rate-limiting is aggressive, I'd suggest 3-5. For Gemini, do not exceed 60.
-----------------------------------------------------------------------------------
-Open AI Settings:
-See https://platform.openai.com/docs/api-reference/chat/create for further details
-----------------------------------------------------------------------------------
-openai_model : ID of the model to use. Kijiku only works with 'chat' models.
-
-openai_system_message : Instructions to the model. Basically tells the model how to translate.
-
-openai_temperature : What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. Lower values are typically better for translation.
-
-openai_top_p : An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered. I generally recommend altering this or temperature but not both.
-
-openai_n : How many chat completion choices to generate for each input message. Do not change this.
-
-openai_stream : If set, partial message deltas will be sent, like in ChatGPT. Tokens will be sent as data-only server-sent events as they become available, with the stream terminated by a data: [DONE] message. See the OpenAI python library on GitHub for example code. Do not change this.
-
-openai_stop : Up to 4 sequences where the API will stop generating further tokens. Do not change this.
-
-openai_logit_bias : Modifies the likelihood of specified tokens appearing in the completion. Do not change this.
-
-openai_max_tokens :  The maximum number of tokens to generate in the chat completion. The total length of input tokens and generated tokens is limited by the model's context length. I wouldn't recommend changing this. Is none by default. If you change to an integer, make sure it doesn't exceed that model's context length or your request will fail and repeat till timeout.
-
-openai_presence_penalty : Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far, increasing the model's likelihood to talk about new topics. While negative values encourage repetition. Should leave this at 0.0.
-
-openai_frequency_penalty : Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model's likelihood to repeat the same line verbatim. Negative values encourage repetition. Should leave this at 0.0.
-----------------------------------------------------------------------------------
-openai_stream, openai_logit_bias, openai_stop and openai_n are included for completion's sake, current versions of Kudasai will hardcode their values when validating the Kijiku_rule.json to their default values. As different values for these settings do not have a use case in Kudasai's current implementation.
-----------------------------------------------------------------------------------
-Gemini Settings:
-https://ai.google.dev/docs/concepts#model-parameters for further details
-----------------------------------------------------------------------------------
-gemini_model : The model to use. Currently only supports gemini-pro and gemini-pro-vision, the 1.0 model and it's aliases.
-
-gemini_prompt : Instructions to the model. Basically tells the model how to translate.
-
-gemini_temperature : What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. Lower values are typically better for translation.
-
-gemini_top_p : An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered. I generally recommend altering this or temperature but not both.
-
-gemini_top_k : Determines the number of most probable tokens to consider for each selection step. A higher value increases diversity, a lower value makes the output more deterministic.
-
-gemini_candidate_count : The number of candidates to generate for each input message. Do not change this.
-
-gemini_stream : If set, partial message deltas will be sent, like in Gemini Chat. Tokens will be sent as data-only server-sent events as they become available, with the stream terminated by a data: [DONE] message. Do not change this.
-
-gemini_stop_sequences : Up to 4 sequences where the API will stop generating further tokens. Do not change this.
-
-gemini_max_output_tokens : The maximum number of tokens to generate in the chat completion. The total length of input tokens and generated tokens is limited by the model's context length. I wouldn't recommend changing this. Is none by default. If you change to an integer, make sure it doesn't exceed that model's context length or your request will fail and repeat till timeout.
-----------------------------------------------------------------------------------
-gemini_stream, gemini_stop_sequences and gemini_candidate_count are included for completion's sake, current versions of Kudasai will hardcode their values when validating the Kijiku_rule.json to their default values. As different values for these settings do not have a use case in Kudasai's current implementation.
-----------------------------------------------------------------------------------
-    """
 
 ##-------------------start-of-validate_json()--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
@@ -99,11 +33,11 @@ gemini_stream, gemini_stop_sequences and gemini_candidate_count are included for
 
         """
 
-        Validates the Kijiku Rules.json file.
+        Validates the translation_settings.json file.
 
         """
 
-        base_kijiku_keys = [
+        base_translation_keys = [
             "prompt_assembly_mode",
             "number_of_lines_per_batch",
             "sentence_fragmenter_mode",
@@ -139,6 +73,13 @@ gemini_stream, gemini_stop_sequences and gemini_candidate_count are included for
             "gemini_max_output_tokens"
         ]
 
+        deepl_keys = [
+            "deepl_context",
+            "deepl_split_sentences",
+            "deepl_preserve_formatting",
+            "deepl_formality"
+        ]
+
         validation_rules = {
             "prompt_assembly_mode": lambda x: isinstance(x, int) and 1 <= x <= 2,
             "number_of_lines_per_batch": lambda x: isinstance(x, int) and x > 0,
@@ -159,31 +100,42 @@ gemini_stream, gemini_stop_sequences and gemini_candidate_count are included for
             "gemini_top_p": lambda x: x is None or (isinstance(x, float) and 0 <= x <= 2),
             "gemini_top_k": lambda x: x is None or (isinstance(x, int) and x >= 0),
             "gemini_max_output_tokens": lambda x: x is None or isinstance(x, int),
+            "deepl_context": lambda x: isinstance(x, str),
+            "deepl_split_sentences": lambda x: isinstance(x, str),
+            "deepl_preserve_formatting": lambda x: isinstance(x, bool),
+            "deepl_formality": lambda x: isinstance(x, str)
         }
         
         try:
             ## ensure categories are present
-            assert "base kijiku settings" in JsonHandler.current_kijiku_rules
-            assert "openai settings" in JsonHandler.current_kijiku_rules
-            assert "gemini settings" in JsonHandler.current_kijiku_rules
+            assert "base translation settings" in JsonHandler.current_translation_settings, "base translation settings not found"
+            assert "openai settings" in JsonHandler.current_translation_settings, "openai settings not found"
+            assert "gemini settings" in JsonHandler.current_translation_settings, "gemini settings not found"
+            assert "deepl settings" in JsonHandler.current_translation_settings, "deepl settings not found"
 
             ## assign to variables to reduce repetitive access
-            base_kijiku_settings = JsonHandler.current_kijiku_rules["base kijiku settings"]
-            openai_settings = JsonHandler.current_kijiku_rules["openai settings"]
-            gemini_settings = JsonHandler.current_kijiku_rules["gemini settings"]
+            base_translation_settings = JsonHandler.current_translation_settings["base translation settings"]
+            openai_settings = JsonHandler.current_translation_settings["openai settings"]
+            gemini_settings = JsonHandler.current_translation_settings["gemini settings"]
+            deepl_settings = JsonHandler.current_translation_settings["deepl settings"]
 
             ## ensure all keys are present
-            ## ensure all keys are present
-            assert all(key in base_kijiku_settings for key in base_kijiku_keys)
-            assert all(key in openai_settings for key in openai_keys)
-            assert all(key in gemini_settings for key in gemini_keys)
+            assert all(key in base_translation_settings for key in base_translation_keys), "base translation settings keys missing"
+            assert all(key in openai_settings for key in openai_keys), "openai settings keys missing"
+            assert all(key in gemini_settings for key in gemini_keys), "gemini settings keys missing"
+            assert all(key in deepl_settings for key in deepl_keys), "deepl settings keys missing"
 
             ## validate each key using the validation rules
             for key, validate in validation_rules.items():
-                if(key in openai_settings and not validate(openai_settings[key])):
+                if(key in base_translation_settings and not validate(base_translation_settings[key])):
+                    raise ValueError(f"Invalid value for {key}")
+                elif(key in openai_settings and not validate(openai_settings[key])):
                     raise ValueError(f"Invalid value for {key}")
                 elif(key in gemini_settings and not validate(gemini_settings[key])):
                     raise ValueError(f"Invalid value for {key}")
+                elif(key in deepl_settings and not validate(deepl_settings[key])):
+                    raise ValueError(f"Invalid value for {key}")
+                
 
             ## force stop/logit_bias into None
             openai_settings["openai_stop"] = None
@@ -195,108 +147,111 @@ gemini_stream, gemini_stop_sequences and gemini_candidate_count are included for
 
             ## force n and candidate_count to 1
             openai_settings["openai_n"] = 1
-
             gemini_settings["gemini_candidate_count"] = 1
 
+            ## ensure deepl_formality and deepl_split_sentences are in allowed values
+            if(isinstance(deepl_settings["deepl_formality"], str) and deepl_settings["deepl_formality"] not in ["default", "more", "less", "prefer_more", "prefer_less"]):
+                raise ValueError("Invalid value for deepl_formality")
+            
+            if(isinstance(deepl_settings["deepl_split_sentences"], str) and deepl_settings["deepl_split_sentences"] not in ["OFF", "ALL", "NO_NEWLINES"]):
+                raise ValueError("Invalid value for deepl_split_sentences")
+
         except Exception as e:
-            Logger.log_action("Kijiku Rules.json is not valid, setting to invalid_placeholder, current:")
-            Logger.log_action("Reason: " + str(e))
-            Logger.log_action(str(JsonHandler.current_kijiku_rules))
-            JsonHandler.current_kijiku_rules = FileEnsurer.INVALID_KIJIKU_RULES_PLACEHOLDER
+            logging.warning(f"translation_settings.json is not valid, setting to invalid_placeholder, current:"
+                            f"\n{JsonHandler.current_translation_settings}"
+                            f"\nReason: {e}")
+            
+            JsonHandler.current_translation_settings = FileEnsurer.INVALID_TRANSLATION_SETTINGS_PLACEHOLDER
 
-        Logger.log_action("Kijiku Rules.json is valid, current:")
-        Logger.log_action(str(JsonHandler.current_kijiku_rules))
+        logging.debug(f"translation_settings.json is valid, current:"
+                    f"\n{JsonHandler.current_translation_settings}")    
 
-##-------------------start-of-reset_kijiku_rules_to_default()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+##-------------------start-of-reset_translation_settings_to_default()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
     @staticmethod
-    def reset_kijiku_rules_to_default() -> None:
+    def reset_translation_settings_to_default() -> None:
 
         """
 
-        Resets the kijiku_rules json to default.
+        Resets the translation_settings.json to default.
 
         """
 
-        JsonHandler.current_kijiku_rules = FileEnsurer.DEFAULT_KIJIKU_RULES
+        JsonHandler.current_translation_settings = FileEnsurer.DEFAULT_TRANSLATION_SETTING
         
-        JsonHandler.dump_kijiku_rules()
+        JsonHandler.dump_translation_settings()
 
-        JsonHandler.load_kijiku_rules()
+        JsonHandler.load_translation_settings()
 
-##-------------------start-of-dump_kijiku_rules()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+##-------------------start-of-dump_translation_settings()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
     @staticmethod
-    def dump_kijiku_rules() -> None:
+    def dump_translation_settings() -> None:
 
         """
 
-        Dumps the Kijiku Rules.json file to disk.
+        Dumps the translation_settings.json file to disk.
 
         """
 
-        with open(FileEnsurer.config_kijiku_rules_path, 'w+', encoding='utf-8') as file:
-            json.dump(JsonHandler.current_kijiku_rules, file)
+        with open(FileEnsurer.config_translation_settings_path, 'w+', encoding='utf-8') as file:
+            json.dump(JsonHandler.current_translation_settings, file)
 
-##-------------------start-of-load_kijiku_rules()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+##-------------------start-of-load_translation_settings()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
     @staticmethod
-    def load_kijiku_rules() -> None:
+    def load_translation_settings() -> None:
 
         """
 
-        Loads the Kijiku Rules.json file into memory.
+        Loads the translation_settings.json file into memory.
 
         """
 
-        with open(FileEnsurer.config_kijiku_rules_path, 'r', encoding='utf-8') as file:
-            JsonHandler.current_kijiku_rules = json.load(file)
+        with open(FileEnsurer.config_translation_settings_path, 'r', encoding='utf-8') as file:
+            JsonHandler.current_translation_settings = json.load(file)
 
-##-------------------start-of-print_kijiku_rules()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+##-------------------start-of-log_translation_settings()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
     @staticmethod
-    def print_kijiku_rules(output:bool=False) -> None:
+    def log_translation_settings(output_to_console:bool=False, specific_section:str | None = None) -> None:
 
         """
 
-        Prints the Kijiku Rules.json file to the log.
+        Prints the translation_settings.json file to the log.
         Logs by default, but can be set to print to console as well.
 
         Parameters:
-        output (bool | optional | default=False) : Whether to print to console as well.
+        output_to_console (bool | optional | default=False) : Whether to print to console as well.
 
         """
         
-        print("-------------------")
-        print("Base Kijiku Settings:")
-        print("-------------------")
-
-        for key,value in JsonHandler.current_kijiku_rules["base kijiku settings"].items():
-            Logger.log_action(key + " : " + str(value), output=output, omit_timestamp=output)
-
-        print("-------------------")
-        print("Open AI Settings:")
-        print("-------------------")
-
-        for key,value in JsonHandler.current_kijiku_rules["openai settings"].items():
-            Logger.log_action(key + " : " + str(value), output=output, omit_timestamp=output)
-
-        print("-------------------")
-        print("Gemini Settings:")
-        print("-------------------")
-
-        for key,value in JsonHandler.current_kijiku_rules["gemini settings"].items():
-            Logger.log_action(key + " : " + str(value), output=output, omit_timestamp=output)
-
+        sections = ["base translation settings", "openai settings", "gemini settings", "deepl settings"]
+        
+        ## if a specific section is provided, only print that section and base translation settings
+        if(specific_section is not None):
+            specific_section = specific_section.lower()
+            sections = [section for section in sections if section.lower() == specific_section or section == "base translation settings"]
+
+        for section in sections:
+            print("-------------------")
+            print(f"{section.capitalize()}:")
+            print("-------------------")
+        
+            for key, value in JsonHandler.current_translation_settings.get(section, {}).items():
+                log_message = f"{key} : {value}"
+                logging.debug(log_message)
+                if(output_to_console):
+                    print(log_message)
 
-##-------------------start-of-change_kijiku_settings()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+##-------------------start-of-change_translation_settings()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
     @staticmethod
-    def change_kijiku_settings() -> None:
+    def change_translation_settings() -> None:
 
         """
 
-        Allows the user to change the settings of the Kijiku Rules.json file
+        Allows the user to change the settings of the translation_settings.json file
 
         """
         
@@ -304,7 +259,7 @@ gemini_stream, gemini_stop_sequences and gemini_candidate_count are included for
 
             Toolkit.clear_console()
 
-            settings_print_message = JsonHandler.kijiku_settings_message + SettingsChanger.generate_settings_change_menu()
+            settings_print_message = JsonHandler.translation_settings_message + SettingsChanger.generate_settings_change_menu()
 
             action = input(settings_print_message).lower()
 
@@ -317,23 +272,26 @@ gemini_stream, gemini_stop_sequences and gemini_candidate_count are included for
 
             elif(action == "d"):
                 print("Resetting to default settings.")
-                JsonHandler.reset_kijiku_rules_to_default()
+                JsonHandler.reset_translation_settings_to_default()
 
-            elif(action in JsonHandler.current_kijiku_rules["base kijiku settings"]):
-                SettingsChanger.change_setting("base kijiku settings", action)
+            elif(action in JsonHandler.current_translation_settings["base translation settings"]):
+                SettingsChanger.change_setting("base translation settings", action)
 
-            elif(action in JsonHandler.current_kijiku_rules["openai settings"]):
+            elif(action in JsonHandler.current_translation_settings["openai settings"]):
                 SettingsChanger.change_setting("openai settings", action)
 
-            elif(action in JsonHandler.current_kijiku_rules["gemini settings"]):
+            elif(action in JsonHandler.current_translation_settings["gemini settings"]):
                 SettingsChanger.change_setting("gemini settings", action)
 
+            elif(action in JsonHandler.current_translation_settings["deepl settings"]):
+                SettingsChanger.change_setting("deepl settings", action)
+
             else:
                 print("Invalid setting name. Please try again.")
 
             Toolkit.pause_console("\nPress enter to continue.")
 
-        JsonHandler.dump_kijiku_rules()
+        JsonHandler.dump_translation_settings()
         
 ##-------------------start-of-convert_to_correct_type()-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
@@ -383,6 +341,10 @@ gemini_stream, gemini_stop_sequences and gemini_candidate_count are included for
             "gemini_stream": {"type": bool, "constraints": lambda x: x is False},
             "gemini_stop_sequences": {"type": None, "constraints": lambda x: x is None},
             "gemini_max_output_tokens": {"type": int, "constraints": lambda x: x is None or isinstance(x, int)},
+            "deepl_context": {"type": str, "constraints": lambda x: isinstance(x, str)},
+            "deepl_split_sentences": {"type": str, "constraints": lambda x: isinstance(x, str)},
+            "deepl_preserve_formatting": {"type": bool, "constraints": lambda x: isinstance(x, bool)},
+            "deepl_formality": {"type": str, "constraints": lambda x: isinstance(x, str)}
         }
 
         if(setting_name not in type_expectations):
@@ -424,7 +386,7 @@ class SettingsChanger:
 
     """
     
-    Handles changing the settings of the Kijiku Rules.json file.
+    Handles changing the settings of the translation_settings.json file.
 
     """
 
@@ -448,17 +410,22 @@ Current settings:
 
 """
 
-        for key,value in JsonHandler.current_kijiku_rules["base kijiku settings"].items():
+        for key,value in JsonHandler.current_translation_settings["base translation settings"].items():
             menu += key + " : " + str(value) + "\n"
 
         print("\n")
 
-        for key,value in JsonHandler.current_kijiku_rules["openai settings"].items():
+        for key,value in JsonHandler.current_translation_settings["openai settings"].items():
             menu += key + " : " + str(value) + "\n"
 
         print("\n")
 
-        for key,value in JsonHandler.current_kijiku_rules["gemini settings"].items():
+        for key,value in JsonHandler.current_translation_settings["gemini settings"].items():
+            menu += key + " : " + str(value) + "\n"
+            
+        print("\n")
+
+        for key,value in JsonHandler.current_translation_settings["deepl settings"].items():
             menu += key + " : " + str(value) + "\n"
 
         menu += """
@@ -476,37 +443,37 @@ Enter the name of the setting you want to change, type d to reset to default, ty
 
         """
 
-        Loads a custom json into the Kijiku Rules.json file.
+        Loads a custom json into the translation_settings.json file.
 
         """
 
         Toolkit.clear_console()
 
         ## saves old rules in case on invalid json
-        old_kijiku_rules = JsonHandler.current_kijiku_rules
+        old_translation_settings = JsonHandler.current_translation_settings
 
         try:
 
             ## loads the custom json file
-            with open(FileEnsurer.external_kijiku_rules_path, 'r', encoding='utf-8') as file:
-                JsonHandler.current_kijiku_rules = json.load(file) 
+            with open(FileEnsurer.external_translation_settings_path, 'r', encoding='utf-8') as file:
+                JsonHandler.current_translation_settings = json.load(file) 
 
             JsonHandler.validate_json()
 
             ## validate_json() sets a dict to the invalid placeholder if it's invalid, so if it's that, it's invalid
-            assert JsonHandler.current_kijiku_rules != FileEnsurer.INVALID_KIJIKU_RULES_PLACEHOLDER
+            assert JsonHandler.current_translation_settings != FileEnsurer.INVALID_TRANSLATION_SETTINGS_PLACEHOLDER
             
-            JsonHandler.dump_kijiku_rules()
+            JsonHandler.dump_translation_settings()
 
             print("Settings loaded successfully.")
         
         except AssertionError:
             print("Invalid JSON file. Please try again.")
-            JsonHandler.current_kijiku_rules = old_kijiku_rules
+            JsonHandler.current_translation_settings = old_translation_settings
 
         except FileNotFoundError:
-            print("Missing JSON file. Make sure you have a json in the same directory as kudasai.py and that the json is named \"kijiku_rules.json\". Please try again.")
-            JsonHandler.current_kijiku_rules = old_kijiku_rules
+            print("Missing JSON file. Make sure you have a json in the same directory as kudasai.py and that the json is named \"translation_settings.json\". Please try again.")
+            JsonHandler.current_translation_settings = old_translation_settings
 
 ##-------------------start-of-change_setting()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
             
@@ -515,7 +482,7 @@ Enter the name of the setting you want to change, type d to reset to default, ty
 
         """
 
-        Changes the setting of the Kijiku Rules.json file.
+        Changes the setting of the translation_settings.json file.
 
         Parameters:
         setting_area (str) : The area of the setting to change.
@@ -528,7 +495,7 @@ Enter the name of the setting you want to change, type d to reset to default, ty
         try:
             converted_value = JsonHandler.convert_to_correct_type(setting_name, new_value)
 
-            JsonHandler.current_kijiku_rules[setting_area][setting_name] = converted_value
+            JsonHandler.current_translation_settings[setting_area][setting_name] = converted_value
             print(f"Updated {setting_name} to {converted_value}.")
         
         except Exception as e:

kudasai.py

@@ -5,6 +5,8 @@ import json
 import asyncio
 import re
 import typing
+import logging
+import argparse
 
 ## third-party libraries
 from kairyou import Kairyou
@@ -12,14 +14,12 @@ from kairyou import Indexer
 from kairyou.types import NameAndOccurrence
 
 ## custom modules
-from models.kaiseki import Kaiseki 
-from models.kijiku import Kijiku
+from modules.common.translator import Translator
 
 from handlers.json_handler import JsonHandler
 
 from modules.common.toolkit import Toolkit
 from modules.common.file_ensurer import FileEnsurer
-from modules.common.logger import Logger
 
 ##-------------------start-of-Kudasai---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
@@ -35,8 +35,49 @@ class Kudasai:
     
     text_to_preprocess:str
     replacement_json:dict
+    knowledge_base:str
 
     need_to_run_kairyou:bool = True
+    need_to_run_indexer:bool = True
+
+##-------------------start-of-setup_logging()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+
+    @staticmethod
+    def setup_logging() -> None:
+
+        """
+
+        Sets up logging for the Kudasai program.
+
+        """
+
+        ## Debug log setup
+        debug_log_handler = logging.FileHandler(FileEnsurer.debug_log_path, mode='w+', encoding='utf-8')
+        debug_log_handler.setLevel(logging.DEBUG)
+        debug_formatter = logging.Formatter('[%(asctime)s] [%(levelname)s] [%(filename)s] %(message)s', datefmt='%Y-%m-%d %H:%M:%S')
+        debug_log_handler.setFormatter(debug_formatter)
+
+        ## Error log setup
+        error_log_handler = logging.FileHandler(FileEnsurer.error_log_path, mode='w+', encoding='utf-8')
+        error_log_handler.setLevel(logging.WARNING)
+        error_formatter = logging.Formatter('[%(asctime)s] [%(levelname)s] [%(filename)s] %(message)s', datefmt='%Y-%m-%d %H:%M:%S')
+        error_log_handler.setFormatter(error_formatter)
+
+        ## Console handler setup
+        console = logging.StreamHandler()
+        console.setLevel(logging.INFO)
+        console_formatter = logging.Formatter('[%(asctime)s] [%(levelname)s] [%(filename)s] %(message)s', datefmt='%Y-%m-%d %H:%M:%S')
+        console.setFormatter(console_formatter)
+
+        ## Add handlers to the logger
+        logger = logging.getLogger('')
+        logger.setLevel(logging.DEBUG)
+        logger.addHandler(debug_log_handler)
+        logger.addHandler(error_log_handler)
+        logger.addHandler(console)
+
+        ## Ensure only INFO level and above messages are sent to the console
+        console.setLevel(logging.INFO)
 
 ##-------------------start-of-boot()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
@@ -45,7 +86,7 @@ class Kudasai:
 
         """
         
-        Does some logging and sets up the console window, regardless of whether the user is running the CLI, WebGUI, or Console version of Kudasai.
+        Does some logging and sets up the console window, and translator settings, regardless of whether the user is running the CLI, WebGUI, or Console version of Kudasai.
 
         """
 
@@ -53,34 +94,36 @@ class Kudasai:
 
         Toolkit.clear_console()
 
+        ## Need to create the output dir FIRST as logging files are located in the output folder
+        FileEnsurer.standard_create_directory(FileEnsurer.output_dir)
+
+        Kudasai.setup_logging()
+
         FileEnsurer.setup_needed_files()
 
-        Logger.log_barrier()
-        Logger.log_action("Kudasai started")
-        Logger.log_action("Current version: " + Toolkit.CURRENT_VERSION)
-        Logger.log_barrier()
+        logging.debug(f"Kudasai started; Current version : {Toolkit.CURRENT_VERSION}")
 
         try:
 
-            with open(FileEnsurer.config_kijiku_rules_path, "r") as kijiku_rules_file:
-                JsonHandler.current_kijiku_rules = json.load(kijiku_rules_file)
+            with open(FileEnsurer.config_translation_settings_path, "r") as translation_settings:
+                JsonHandler.current_translation_settings = json.load(translation_settings)
 
             JsonHandler.validate_json()
 
-            assert JsonHandler.current_kijiku_rules != FileEnsurer.INVALID_KIJIKU_RULES_PLACEHOLDER
+            assert JsonHandler.current_translation_settings != FileEnsurer.INVALID_TRANSLATION_SETTINGS_PLACEHOLDER
 
         except:
 
-            print("Invalid kijiku_rules.json file. Please check the file for errors. If you are unsure, delete the file and run Kudasai again. Your kijiku rules file is located at: " + FileEnsurer.config_kijiku_rules_path)
+            print("Invalid translation_settings.json file. Please check the file for errors or mistakes. If you are unsure, delete the file and run Kudasai again. Your file is located at: " + FileEnsurer.config_translation_settings_path)
 
             Toolkit.pause_console()
 
-            raise Exception("Invalid kijiku_rules.json file. Please check the file for errors. If you are unsure, delete the file and run Kudasai again. Your kijiku rules file is located at: " + FileEnsurer.config_kijiku_rules_path)
+            raise Exception("Invalid translation_settings.json file. Please check the file for errors or mistakes. If you are unsure, delete the file and run Kudasai again. Your file is located at: " + FileEnsurer.config_translation_settings_path)
         
 ##-------------------start-of-run_kairyou_indexer()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
     @staticmethod
-    def run_kairyou_indexer(text_to_index:str, replacement_json:typing.Union[dict,str]) -> typing.Tuple[str, str]:
+    def run_kairyou_indexer(text_to_index:str, replacement_json:typing.Union[dict,str], knowledge_base:str) -> typing.Tuple[str, str]:
 
         """
 
@@ -98,8 +141,6 @@ class Kudasai:
 
         Toolkit.clear_console()
 
-        knowledge_base = input("Please enter the path to the knowledge base you would like to use for the indexer (can be text, a path to a txt file, or a path to a directory of txt files):\n").strip('"')
-
         ## unique names is a list of named tuples, with the fields name and occurrence
         unique_names, indexing_log = Indexer.index(text_to_index, knowledge_base, replacement_json)
 
@@ -144,7 +185,7 @@ class Kudasai:
                     new_text += text[last_end:match.start()] + f">>>{name}<<<"
                     last_end = match.end()
 
-            new_text += text[last_end:]  # Append the rest of the text
+            new_text += text[last_end:]  ## Append the rest of the text
             text = new_text
 
         return text
@@ -166,8 +207,13 @@ class Kudasai:
 
             indexing_log = ""
 
-            if(Kudasai.replacement_json not in ["", FileEnsurer.blank_rules_path, FileEnsurer.standard_read_json(FileEnsurer.blank_rules_path)] and input("Would you like to use Kairyou's Indexer to index the preprocessed text? (1 for yes, 2 for no)\n") == "1"):
-                Kudasai.text_to_preprocess, indexing_log = Kudasai.run_kairyou_indexer(Kudasai.text_to_preprocess, Kudasai.replacement_json)
+            if(Kudasai.replacement_json not in ["", 
+                                                FileEnsurer.blank_rules_path, 
+                                                FileEnsurer.standard_read_json(FileEnsurer.blank_rules_path)] 
+
+               and Kudasai.need_to_run_indexer
+               and Kudasai.knowledge_base != ""):
+                Kudasai.text_to_preprocess, indexing_log = Kudasai.run_kairyou_indexer(Kudasai.text_to_preprocess, Kudasai.replacement_json, Kudasai.knowledge_base)
 
             preprocessed_text, preprocessing_log, error_log = Kairyou.preprocess(Kudasai.text_to_preprocess, Kudasai.replacement_json)
 
@@ -178,6 +224,9 @@ class Kudasai:
             if(indexing_log != ""):
                 preprocessing_log = indexing_log + "\n\n" + preprocessing_log
 
+            if(preprocessing_log == "Skipped"):
+                preprocessing_log = "Preprocessing skipped."
+
             print(preprocessing_log) 
 
             timestamp = Toolkit.get_timestamp(is_archival=True)
@@ -190,7 +239,7 @@ class Kudasai:
         else:
             print("(Preprocessing skipped)")
 
-        await Kudasai.determine_autotranslation_module()
+        await Kudasai.run_translator()
 
         Toolkit.pause_console("\nPress any key to exit...")
 
@@ -214,92 +263,32 @@ class Kudasai:
             Toolkit.pause_console()
             Toolkit.clear_console()
     
-##-------------------start-of-determine_autotranslation_module()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+##-------------------start-of-run_translator()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
     @staticmethod
-    async def determine_autotranslation_module() -> None:
+    async def run_translator(is_cli:bool=False) -> None:
 
         """
         
-        If the user is running the CLI or Console version of Kudasai, this function is called to determine which autotranslation module to use.
+        If the user is running the CLI or Console version of Kudasai, this function is called to run the Translator module.
 
         """
 
-        if(not Kudasai.connection):
-            Toolkit.clear_console()
-
-            print("You are not connected to the internet. Please connect to the internet to use the autotranslation feature.\n")
-            Toolkit.pause_console()
-
-            exit()
+        Translator.is_cli = is_cli
 
-        pathing = ""
-
-        pathing_msg = "Please select an auto-translation module:\n\n1.Kaiseki (deepL)\n2.Kijiku (OpenAI/Gemini)\n3.Exit\n\n"
-
-        pathing = input(pathing_msg)
+        logging.info("Translator started")
 
         Toolkit.clear_console()
 
-        if(pathing == "1"):
-            Kudasai.run_kaiseki()
-        elif(pathing == "2"):
-            await Kudasai.run_kijiku()
-        else:
-            Toolkit.clear_console()
-            exit()
-
-##-------------------start-of-run_kaiseki()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+        Translator.text_to_translate = [line for line in Kudasai.text_to_preprocess.splitlines()]
 
-    @staticmethod
-    def run_kaiseki() -> None:
-
-        """
-        
-        If the user is running the CLI or Console version of Kudasai, this function is called to run the Kaiseki module.
-
-        """
-
-        Logger.log_action("--------------------")
-        Logger.log_action("Kaiseki started")
-        Logger.log_action("--------------------")
-
-        Kaiseki.text_to_translate = [line for line in Kudasai.text_to_preprocess.splitlines()]
-
-        Kaiseki.translate()
+        await Translator.translate()
 
         Toolkit.clear_console()
 
-        print(Kaiseki.translation_print_result)
-
-        Kaiseki.write_kaiseki_results() 
-
-##-------------------start-of-run_kijiku()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-
-    @staticmethod
-    async def run_kijiku() -> None:
+        print(Translator.translation_print_result)
 
-        """
-        
-        If the user is running the CLI or Console version of Kudasai, this function is called to run the Kijiku module.
-
-        """
-
-        Logger.log_action("--------------------")
-        Logger.log_action("Kijiku started")
-        Logger.log_action("--------------------")
-
-        Toolkit.clear_console()
-
-        Kijiku.text_to_translate = [line for line in Kudasai.text_to_preprocess.splitlines()]
-
-        await Kijiku.translate()
-
-        Toolkit.clear_console()
-
-        print(Kijiku.translation_print_result)
-
-        Kijiku.write_kijiku_results()
+        Translator.write_translator_results()
 
 ##-------------------start-of-main()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
@@ -319,7 +308,7 @@ async def main() -> None:
         if(len(sys.argv) <= 1):
             await run_console_version()
         
-        elif(len(sys.argv) in [2, 3]):
+        elif(len(sys.argv) in [2, 3, 4, 5, 6]):
             await run_cli_version()
 
         else:
@@ -340,21 +329,26 @@ async def run_console_version():
 
     try:
 
-        path_to_text_to_preprocess = input("Please enter the path to the input file to be processed:\n").strip('"')
+        path_to_text_to_preprocess = input("Please enter the path to the input file to be preprocessed/translated:\n").strip('"')
         Kudasai.text_to_preprocess = FileEnsurer.standard_read_file(path_to_text_to_preprocess)
         Toolkit.clear_console()
 
-        path_to_replacement_json = input("Please enter the path to the replacement json file:\n").strip('"')
+        path_to_replacement_json = input("Please enter the path to the replacement json file (Press enter if skipping to translation):\n").strip('"')
         Kudasai.replacement_json = FileEnsurer.standard_read_json(path_to_replacement_json if path_to_replacement_json else FileEnsurer.blank_rules_path)
         Toolkit.clear_console()
 
+        if(path_to_replacement_json != ""):
+            Kudasai.knowledge_base = input("Please enter the path to the knowledge base you would like to use for the name indexer (can be text, a path to a txt file, or a path to a directory of txt files (Press enter if skipping name indexing):\n").strip('"')
+            Toolkit.clear_console()
+
     except Exception as e:
         print_usage_statement()
 
         raise e
+    
+    print("In progress...")
 
     await Kudasai.run_kudasai()
-    Logger.push_batch()
 
 ##-------------------start-of-run_cli_version()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
@@ -366,22 +360,99 @@ async def run_cli_version():
 
     """
 
+    def determine_argument_type(arg:str) -> str:
+
+        """
+
+        Determines the third argument for the CLI version of Kudasai.
+
+        """
+
+        conditions = [
+            (lambda arg: arg in ["deepl", "openai", "gemini"], "translation_method"),
+            (lambda arg: os.path.exists(arg) and not ".json" in arg, "text_to_translate"),
+            (lambda arg: len(arg) > 10 and not os.path.exists(arg), "api_key"),
+            (lambda arg: arg == "translate", "identifier"),
+            (lambda arg: os.path.exists(arg) and ".json" in arg, "translation_settings_json")
+        ]
+
+        for condition, result in conditions:
+            if(condition(arg)):
+                print(result)
+                return result
+
+        raise Exception("Invalid argument. Please use 'deepl', 'openai', or 'gemini'.")
+        
+    mode = ""
+
     try:
 
-        Kudasai.text_to_preprocess = FileEnsurer.standard_read_file(sys.argv[1].strip('"'))
-        Kudasai.replacement_json = FileEnsurer.standard_read_json(sys.argv[2].strip('"') if(len(sys.argv) == 3) else FileEnsurer.blank_rules_path)
+        indices = {
+            "preprocess": {"text_to_preprocess_index": 2, "replacement_json_index": 3, "knowledge_base_index": 4},
+            "translate": {"text_to_translate_index": 2},
+            "--help": {}
+        }
 
-    except Exception as e:
-        print_usage_statement()
+        try:
+            arg_indices = indices[sys.argv[1]]
+            mode = sys.argv[1]
 
-        raise e
+        except KeyError:
+            print_usage_statement()
+            raise Exception("Invalid mode. Please use 'preprocess' or 'translate'. Please use --help for more information.")
+        
+        if(mode == "preprocess"):
+    
+            Kudasai.text_to_preprocess = FileEnsurer.standard_read_file(sys.argv[arg_indices['text_to_preprocess_index']].strip('"'))
+            Kudasai.replacement_json = FileEnsurer.standard_read_json(sys.argv[arg_indices['replacement_json_index']].strip('"')) if len(sys.argv) >= arg_indices['replacement_json_index'] + 1 else FileEnsurer.standard_read_json(FileEnsurer.blank_rules_path)
+            Kudasai.knowledge_base = sys.argv[arg_indices['knowledge_base_index']].strip('"') if len(sys.argv) == arg_indices['knowledge_base_index'] + 1 else ""
+
+            if(len(sys.argv) == 2):
+                Kudasai.need_to_run_kairyou = False
+            elif(len(sys.argv) == 3):
+                Kudasai.need_to_run_indexer = False
+        
+            await Kudasai.run_kudasai()
 
-    if(len(sys.argv) == 2):
-        Kudasai.need_to_run_kairyou = False
+        elif(mode == "translate"):
 
-    await Kudasai.run_kudasai()
-    Logger.push_batch()
+            method_to_translation_mode = {
+                "openai": "1",
+                "gemini": "2",
+                "deepl": "3"
+            }
+
+            Kudasai.text_to_preprocess = FileEnsurer.standard_read_file(sys.argv[arg_indices['text_to_translate_index']].strip('"'))
+            
+            sys.argv.pop(0)
+
+            arg_dict = {arg.strip('"'): determine_argument_type(arg.strip('"')) for arg in sys.argv}
+
+            assert len(arg_dict) == len(set(arg_dict)), "Invalid arguments. Please use --help for more information."
+
+            arg_type_action_map = {
+                "translation_method": lambda arg: setattr(Translator, 'TRANSLATION_METHOD', method_to_translation_mode[arg]),
+                "translation_settings_json": lambda arg: setattr(JsonHandler, 'current_translation_settings', FileEnsurer.standard_read_json(arg)),
+                "api_key": lambda arg: setattr(Translator, 'pre_provided_api_key', arg),
+                "identifier": lambda arg: None,
+                "text_to_translate": lambda arg: setattr(Kudasai, 'text_to_preprocess', FileEnsurer.standard_read_file(arg))
+            }
 
+            for arg, arg_type in arg_dict.items():
+                if(arg_type in arg_type_action_map):
+                    arg_type_action_map[arg_type](arg)
+                else:
+                    raise Exception("Invalid argument type. Please use --help for more information.")
+
+            await Kudasai.run_translator(is_cli=True)
+
+        else:
+            print_usage_statement() 
+
+    except Exception as e:
+        print_usage_statement()
+        raise e
+    
 ##-------------------start-of-print_usage_statement()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
 def print_usage_statement():
@@ -391,14 +462,47 @@ def print_usage_statement():
     Prints the usage statement for the CLI version of Kudasai.
 
     """
+    python_command = "python" if Toolkit.is_windows() else "python3"
+
+    print(f"""
+Usage: {python_command} Kudasai.py <mode> <required_arguments> [optional_arguments]
+
+Modes:
+  preprocess
+    Preprocesses the text file using the provided replacement JSON.
+
+    Required arguments:
+      <input_file>              Path to the text file to preprocess. This a path to a text file
+      <replacement_json>        Path to the replacement JSON file. This is a path to a json file.
+
+    Optional arguments:
+      <knowledge_base>          Path to the knowledge base file. This can be either a directory, file, or even text.
+
+    Example:
+      {python_command} Kudasai.py preprocess "C:\\path\\to\\input_file.txt" "C:\\path\\to\\replacement_json.json" "C:\\path\\to\\knowledge_base"
+
+  translate
+    Translates the text file using the specified translation method.
+
+    Required arguments:
+      <input_file>              Path to the text file to translate. This is a txt file.
+
+    Optional arguments:
+      <translation_method>      Translation method to use ('deepl', 'openai', or 'gemini'). This defaults to deepl
+      <translation_settings_json> Path to the translation settings JSON file. This will override the current loaded settings.
+      <api_key>                  API key for the translation service. If not provided, it will use the one on file, otherwise it will ask if not provided
+
+    Example:
+      {python_command} Kudasai.py translate "C:\\path\\to\\input_file.txt" gemini "C:\\path\\to\\translation_settings.json" "YOUR API KEY"
 
-    Logger.log_action("Usage: python Kudasai.py <input_file> <replacement_json>", output=True, omit_timestamp=True)
-    Logger.log_action("or run Kudasai.py without any arguments to run the console version.", output=True, omit_timestamp=True)
+Additional Notes:
+- All arguments should be enclosed in double quotes if they contain spaces. But double quotes are optional and will be striped. Single quotes are not allowed
+- For more information, refer to the documentation at README.md
+""")
 
-    print("\n")
 
 ##-------------------start-of-submain()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
 
-if(__name__ == '__main__'):
+if(__name__ == "__main__"):
     asyncio.run(main())

lib/common/translation_settings_description.txt

@@ -0,0 +1,79 @@
+----------------------------------------------------------------------------------
+Base Translation Settings:
+
+prompt_assembly_mode : 1 or 2. 1 means the system message will actually be treated as a system message. 2 means it'll be treated as a user message. 1 is recommend for gpt-4 otherwise either works. For Gemini & DeepL, this setting is ignored.
+
+number_of_lines_per_batch : The number of lines to be built into a prompt at once. Theoretically, more lines would be more cost effective, but other complications may occur with higher lines. So far been tested up to 48 by me.
+
+sentence_fragmenter_mode : 1 or 2  (1 - via regex and other nonsense) 2 - None (Takes formatting and text directly from API return)) the API can sometimes return a result on a single line, so this determines the way Kudasai fragments the sentences if at all. Use 2 for newer models and Deepl.
+
+je_check_mode : 1 or 2, 1 will print out the jap then the english below separated by ---, 2 will attempt to pair the english and jap sentences, placing the jap above the eng. If it cannot, it will default to 1. Use 2 for newer models and DeepL.
+
+number_of_malformed_batch_retries : (Malformed batch is when je-fixing fails) How many times Kudasai will attempt to mend a malformed batch (mending is resending the request). Be careful with increasing as cost increases at (cost * length * n) at worst case. This setting is ignored if je_check_mode is set to 1.
+
+batch_retry_timeout : How long Kudasai will try to translate a batch in seconds, if a requests exceeds this duration, Kudasai will leave it untranslated.
+
+number_of_concurrent_batches : How many translations batches Kudasai will send to the translation API at a time. For OpenAI, be conservative as rate-limiting is aggressive, I'd suggest 3-5. For Gemini, do not exceed 15 for 1.0 or 2 for 1.5. This setting more or less doesn't matter for DeepL.
+----------------------------------------------------------------------------------
+Open AI Settings:
+See https://platform.openai.com/docs/api-reference/chat/create for further details
+----------------------------------------------------------------------------------
+openai_model : ID of the model to use. Kudasai only works with 'chat' models.
+
+openai_system_message : Instructions to the model. Basically tells the model how to translate.
+
+openai_temperature : What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. Lower values are typically better for translation.
+
+openai_top_p : An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered. I generally recommend altering this or temperature but not both.
+
+openai_n : How many chat completion choices to generate for each input message. Do not change this, as Kudasai will always use 1.
+
+openai_stream : If set, partial message deltas will be sent, like in ChatGPT. Tokens will be sent as data-only server-sent events as they become available, with the stream terminated by a data: [DONE] message. See the OpenAI python library on GitHub for example code. Do not change this as Kudasai does not support this feature.
+
+openai_stop : Up to 4 sequences where the API will stop generating further tokens. Do not change this as Kudasai does not support this feature.
+
+openai_logit_bias : Modifies the likelihood of specified tokens appearing in the completion. Do not change this as Kudasai does not support this feature.
+
+openai_max_tokens :  The maximum number of tokens to generate in the chat completion. The total length of input tokens and generated tokens is limited by the model's context length. I wouldn't recommend changing this. Is none by default. If you change to an integer, make sure it doesn't exceed that model's context length or your request will fail and repeat till timeout.
+
+openai_presence_penalty : Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far, increasing the model's likelihood to talk about new topics. While negative values encourage repetition. Should leave this at 0.0.
+
+openai_frequency_penalty : Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model's likelihood to repeat the same line verbatim. Negative values encourage repetition. Should leave this at 0.0.
+----------------------------------------------------------------------------------
+openai_stream, openai_logit_bias, openai_stop and openai_n are included for completion's sake, current versions of Kudasai will hardcode their values when validating the translation_settings.json to their default values. As different values for these settings do not have a use case in Kudasai's current implementation.
+----------------------------------------------------------------------------------
+Gemini Settings:
+See https://ai.google.dev/docs/concepts#model-parameters for further details
+----------------------------------------------------------------------------------
+gemini_model : The model to use. Currently only supports gemini-pro and gemini-pro-vision, the 1.0 model and 1.5 models and their aliases.
+
+gemini_prompt : Instructions to the model. Basically tells the model how to translate.
+
+gemini_temperature : What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. Lower values are typically better for translation.
+
+gemini_top_p : An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered. I generally recommend altering this or temperature but not both.
+
+gemini_top_k : Determines the number of most probable tokens to consider for each selection step. A higher value increases diversity, a lower value makes the output more deterministic.
+
+gemini_candidate_count : The number of candidates to generate for each input message. Do not change this as Kudasai will always use 1.
+
+gemini_stream : If set, partial message deltas will be sent, like in Gemini Chat. Tokens will be sent as data-only server-sent events as they become available, with the stream terminated by a data: [DONE] message. Do not change this as Kudasai does not support this feature.
+
+gemini_stop_sequences : Up to 4 sequences where the API will stop generating further tokens. Do not change this as Kudasai does not support this feature.
+
+gemini_max_output_tokens : The maximum number of tokens to generate in the chat completion. The total length of input tokens and generated tokens is limited by the model's context length. I wouldn't recommend changing this. Is none by default. If you change to an integer, make sure it doesn't exceed that model's context length or your request will fail and repeat till timeout.
+----------------------------------------------------------------------------------
+gemini_stream, gemini_stop_sequences and gemini_candidate_count are included for completion's sake, current versions of Kudasai will hardcode their values when validating the translation_settings.json to their default values. As different values for these settings do not have a use case in Kudasai's current implementation.
+----------------------------------------------------------------------------------
+Deepl Settings:
+See https://developers.deepl.com/docs/api-reference/translate for further details
+----------------------------------------------------------------------------------
+deepl_context : The context in which the text should be translated. This is used to improve the translation. If you don't have any context, you can leave this empty. This is a DeepL Alpha feature and could be subject to change.
+
+deepl_split_sentences : How the text should be split into sentences. Possible values are 'OFF', 'ALL', 'NO_NEWLINES'.
+
+deepl_preserve_formatting : Whether the formatting of the text should be preserved. If you don't want to preserve the formatting, you can set this to False. Otherwise, set it to True.
+
+deepl_formality : The formality of the text. Possible values are 'default', 'more', 'less', 'prefer_more', 'prefer_less'.
+
+----------------------------------------------------------------------------------

lib/gui/HUGGING_FACE_README.md

@@ -0,0 +1,224 @@
+---
+license: gpl-3.0
+title: Kudasai
+sdk: gradio
+emoji: 🈷️
+python_version: 3.10.0
+app_file: webgui.py
+colorFrom: gray
+colorTo: gray
+short_description: Japanese-English preprocessor with automated translation.
+pinned: true
+---
+
+---------------------------------------------------------------------------------------------------------------------------------------------------
+**Table of Contents**
+
+- [**Notes**](#notes)
+- [**General Usage**](#general-usage)
+- [**Indexing and Preprocessing**](#indexing-and-preprocessing)
+- [**Translator**](#translator)
+- [**Translator Settings**](#translator-settings)
+- [**Web GUI**](#web-gui)
+- [**License**](#license)
+- [**Contact**](#contact)
+- [**Acknowledgements**](#acknowledgements)
+
+---------------------------------------------------------------------------------------------------------------------------------------------------
+## **Notes**<a name="notes"></a>
+
+This readme is for the Hugging Space instance of Kudasai's WebGUI and the WebGUI itself, to run Kudasai locally or see any info on the project, please see the [GitHub Page](https://github.com/Bikatr7/Kudasai).
+
+Streamlining Japanese-English Translation with Advanced Preprocessing and Integrated Translation Technologies.
+
+Preprocessor and Translation logic is sourced from external packages, which I also designed, see [Kairyou](https://github.com/Bikatr7/Kairyou) and [EasyTL](https://github.com/Bikatr7/easytl) for more information.
+
+Kudasai has a public trello board, you can find it [here](https://trello.com/b/Wsuwr24S/kudasai) to see what I'm working on and what's coming up.
+
+The WebGUI on huggingface does not save anything through runs, so you will need to download the output files or copy the text out of the webgui. API keys are not saved, and the output folder is overwritten every time you run it. Archives deleted every run as well.
+
+Kudasai is proud to have been a Backdrop Build v3 Finalist:
+https://backdropbuild.com/builds/v3/kudasai
+
+---------------------------------------------------------------------------------------------------------------------------------------------------
+
+## **General Usage**<a name="general-usage"></a>
+
+Kudasai's WebGUI is pretty easy to understand for the general usage, most incorrect actions will be caught by the system and a message will be displayed to the user on how to correct it.
+
+Normally, Kudasai would save files to the local system, but on Hugging Face's servers, this is not possible. Instead, you'll have to click the 'Save As' button to download the files to your local system.
+
+Or you can click the copy button on the top right of textbox modals to copy the text to your clipboard.
+
+For further details, see below chapters.
+
+---------------------------------------------------------------------------------------------------------------------------------------------------
+
+## **Indexing and Preprocessing**<a name="kairyou"></a>
+
+This section can be skipped if you're only interested in translation or do not know what indexing or preprocessing is.
+
+Indexing is not for everyone, only use it if you have a large amount of previous text and want to flag new names. It can be a very slow and long process, especially on Hugging Face's servers. It's recommended to use a local version of Kudasai for this process.
+
+You'll need a txt file or some text to index. You'll also need a knowledge base, this can either be a single txt file or a directory of them, as well as a replacements json. Either Kudasai or Fukuin Type works. See [this](https://github.com/Bikatr7/Kairyou?tab=readme-ov-file#kairyou) for further details on replacement jsons.
+
+Please do indexing before preprocessing, output is neater that way.
+
+For Preprocessing, you'll need a txt file or some text to preprocess. You'll also need a replacements json. Either Kudasai or Fukuin Type works like with indexing.
+
+For both, text is put in the textbox modals, with the output text being in the first field, and results being in the second field. 
+
+They both have a debug field, but neither module really uses it.
+
+---------------------------------------------------------------------------------------------------------------------------------------------------
+
+## **Translator**<a name="translator"></a>
+
+Kudasai supports 3 different translation methods at the moment, OpenAI's GPT, Google's Gemini, and DeepL. 
+
+For OpenAI, you'll need an API key, you can get one [here](https://platform.openai.com/docs/api-reference/authentication). This is a paid service with no free tier.
+
+For Gemini, you'll also need an API key, you can get one [here](https://ai.google.dev/tutorials/setup). Gemini is free to use under a certain limit, 2 RPM for 1.5 and 15 RPM for 1.0.
+
+For DeepL, you'll need an API key too, you can get one [here](https://www.deepl.com/pro#developer). DeepL is also a paid service but is free under 500k characters a month.
+
+I'd recommend using GPT for most things, as it's generally better at translation.
+
+Mostly straightforward, choose your translation method, fill in your API key, and select your text. You'll also need to add your settings file if on HuggingFace if you want to tune the output, but the default is generally fine.
+
+You can calculate costs here or just translate. Output will show in the appropriate fields.
+
+For further details on the settings file, see [here](#translation-with-llms-settings).
+
+---------------------------------------------------------------------------------------------------------------------------------------------------
+
+## **Translator Settings**<a name="translator-settings"></a>
+
+(Fairly technical, can be abstracted away by using default settings or someone else's settings file.)
+
+    Base Translation Settings:
+
+    prompt_assembly_mode : 1 or 2. 1 means the system message will actually be treated as a system message. 2 means it'll be treated as a user message. 1 is recommend for gpt-4 otherwise either works. For Gemini & DeepL, this setting is ignored.
+
+    number_of_lines_per_batch : The number of lines to be built into a prompt at once. Theoretically, more lines would be more cost effective, but other complications may occur with higher lines. So far been tested up to 48 by me.
+
+    sentence_fragmenter_mode : 1 or 2  (1 - via regex and other nonsense) 2 - None (Takes formatting and text directly from API return)) the API can sometimes return a result on a single line, so this determines the way Kudasai fragments the sentences if at all. Use 2 for newer models and Deepl.
+
+    je_check_mode : 1 or 2, 1 will print out the jap then the english below separated by ---, 2 will attempt to pair the english and jap sentences, placing the jap above the eng. If it cannot, it will default to 1. Use 2 for newer models and DeepL.
+
+    number_of_malformed_batch_retries : (Malformed batch is when je-fixing fails) How many times Kudasai will attempt to mend a malformed batch (mending is resending the request). Be careful with increasing as cost increases at (cost * length * n) at worst case. This setting is ignored if je_check_mode is set to 1.
+
+    batch_retry_timeout : How long Kudasai will try to translate a batch in seconds, if a requests exceeds this duration, Kudasai will leave it untranslated.
+
+    number_of_concurrent_batches : How many translations batches Kudasai will send to the translation API at a time. For OpenAI, be conservative as rate-limiting is aggressive, I'd suggest 3-5. For Gemini, do not exceed 15 for 1.0 or 2 for 1.5. This setting more or less doesn't matter for DeepL.
+    ----------------------------------------------------------------------------------
+    Open AI Settings:
+    See https://platform.openai.com/docs/api-reference/chat/create for further details
+    ----------------------------------------------------------------------------------
+    openai_model : ID of the model to use. Kudasai only works with 'chat' models.
+
+    openai_system_message : Instructions to the model. Basically tells the model how to translate.
+
+    openai_temperature : What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. Lower values are typically better for translation.
+
+    openai_top_p : An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered. I generally recommend altering this or temperature but not both.
+
+    openai_n : How many chat completion choices to generate for each input message. Do not change this, as Kudasai will always use 1.
+
+    openai_stream : If set, partial message deltas will be sent, like in ChatGPT. Tokens will be sent as data-only server-sent events as they become available, with the stream terminated by a data: [DONE] message. See the OpenAI python library on GitHub for example code. Do not change this as Kudasai does not support this feature.
+
+    openai_stop : Up to 4 sequences where the API will stop generating further tokens. Do not change this as Kudasai does not support this feature.
+
+    openai_logit_bias : Modifies the likelihood of specified tokens appearing in the completion. Do not change this as Kudasai does not support this feature.
+
+    openai_max_tokens :  The maximum number of tokens to generate in the chat completion. The total length of input tokens and generated tokens is limited by the model's context length. I wouldn't recommend changing this. Is none by default. If you change to an integer, make sure it doesn't exceed that model's context length or your request will fail and repeat till timeout.
+
+    openai_presence_penalty : Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far, increasing the model's likelihood to talk about new topics. While negative values encourage repetition. Should leave this at 0.0.
+
+    openai_frequency_penalty : Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model's likelihood to repeat the same line verbatim. Negative values encourage repetition. Should leave this at 0.0.
+    ----------------------------------------------------------------------------------
+    openai_stream, openai_logit_bias, openai_stop and openai_n are included for completion's sake, current versions of Kudasai will hardcode their values when validating the translation_settings.json to their default values. As different values for these settings do not have a use case in Kudasai's current implementation.
+    ----------------------------------------------------------------------------------
+    Gemini Settings:
+    See https://ai.google.dev/docs/concepts#model-parameters for further details
+    ----------------------------------------------------------------------------------
+    gemini_model : The model to use. Currently only supports gemini-pro and gemini-pro-vision, the 1.0 model and 1.5 models and their aliases.
+
+    gemini_prompt : Instructions to the model. Basically tells the model how to translate.
+
+    gemini_temperature : What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. Lower values are typically better for translation.
+
+    gemini_top_p : An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered. I generally recommend altering this or temperature but not both.
+
+    gemini_top_k : Determines the number of most probable tokens to consider for each selection step. A higher value increases diversity, a lower value makes the output more deterministic.
+
+    gemini_candidate_count : The number of candidates to generate for each input message. Do not change this as Kudasai will always use 1.
+
+    gemini_stream : If set, partial message deltas will be sent, like in Gemini Chat. Tokens will be sent as data-only server-sent events as they become available, with the stream terminated by a data: [DONE] message. Do not change this as Kudasai does not support this feature.
+
+    gemini_stop_sequences : Up to 4 sequences where the API will stop generating further tokens. Do not change this as Kudasai does not support this feature.
+
+    gemini_max_output_tokens : The maximum number of tokens to generate in the chat completion. The total length of input tokens and generated tokens is limited by the model's context length. I wouldn't recommend changing this. Is none by default. If you change to an integer, make sure it doesn't exceed that model's context length or your request will fail and repeat till timeout.
+    ----------------------------------------------------------------------------------
+    gemini_stream, gemini_stop_sequences and gemini_candidate_count are included for completion's sake, current versions of Kudasai will hardcode their values when validating the translation_settings.json to their default values. As different values for these settings do not have a use case in Kudasai's current implementation.
+    ----------------------------------------------------------------------------------
+    Deepl Settings:
+    See https://developers.deepl.com/docs/api-reference/translate for further details
+    ----------------------------------------------------------------------------------
+    deepl_context : The context in which the text should be translated. This is used to improve the translation. If you don't have any context, you can leave this empty. This is a DeepL Alpha feature and could be subject to change.
+
+    deepl_split_sentences : How the text should be split into sentences. Possible values are 'OFF', 'ALL', 'NO_NEWLINES'.
+
+    deepl_preserve_formatting : Whether the formatting of the text should be preserved. If you don't want to preserve the formatting, you can set this to False. Otherwise, set it to True.
+
+    deepl_formality : The formality of the text. Possible values are 'default', 'more', 'less', 'prefer_more', 'prefer_less'.
+
+---------------------------------------------------------------------------------------------------------------------------------------------------
+
+## **Web GUI**<a name="webgui"></a>
+
+Below are some images of the Web GUI.
+
+Name Indexing | Kairyou:
+![Name Indexing Screen | Kairyou](https://i.imgur.com/QCPqjrw.jpeg)
+
+Text Preprocessing | Kairyou:
+![Text Preprocessing Screen | Kairyou](https://i.imgur.com/r8nHEvw.jpeg)
+
+Text Translation | Translator:
+![Text Translation Screen | Translator](https://i.imgur.com/0E9q2eh.jpeg)
+
+Translation Settings Page 1:
+![Translation Settings Page 1](https://i.imgur.com/0E9q2eh.jpeg)
+
+Translation Settings Page 2:
+![Translation Settings Page 2](https://i.imgur.com/8MQk6pL.jpeg)
+
+Logging Page:
+![Logging Page](https://i.imgur.com/vDPCUQC.jpeg)
+
+---------------------------------------------------------------------------------------------------------------------------------------------------
+## **License**<a name="license"></a>
+
+This project (Kudasai) is licensed under the GNU General Public License (GPL). You can find the full text of the license in the [LICENSE](License.md) file.
+
+The GPL is a copyleft license that promotes the principles of open-source software. It ensures that any derivative works based on this project must also be distributed under the same GPL license. This license grants you the freedom to use, modify, and distribute the software.
+
+Please note that this information is a brief summary of the GPL. For a detailed understanding of your rights and obligations under this license, please refer to the full license text.
+
+---------------------------------------------------------------------------------------------------------------------------------------------------
+## **Contact**<a name="contact"></a>
+
+If you have any questions, comments, or concerns, please feel free to contact me at [Bikatr7@proton.me](mailto:Bikatr7@proton.me)
+
+For any bugs or suggestions please use the issues tab [here](https://github.com/Bikatr7/Kudasai/issues).
+
+I actively encourage and welcome any feedback on this project.
+
+---------------------------------------------------------------------------------------------------------------------------------------------------
+
+## **Acknowledgements**<a name="acknowledgements"></a>
+
+Kudasai gets it's original name idea from it's inspiration, Atreyagaurav's Onegai. Which also means please. You can find that [here](https://github.com/Atreyagaurav/onegai)
+
+---------------------------------------------------------------------------------------------------------------------------------------------------

lib/gui/save_to_file.js

@@ -0,0 +1,9 @@
+(text) => {
+    const blob = new Blob([text], { type: 'text/plain;charset=utf-8' });
+    const url = URL.createObjectURL(blob);
+    const a = document.createElement('a');
+    a.href = url;
+    a.download = 'downloaded_text.txt';
+    a.click();
+    URL.revokeObjectURL(url);
+}

models/kaiseki.py

@@ -1,583 +0,0 @@
-## Basically Deprecated, use Kijiku instead. Currently only maintained for backwards compatibility.
-##---------------------------------------
-##---------------------------------------
-## built-in libraries
-import string
-import time
-import re
-import base64
-import time
-
-## third-party libraries
-from easytl import EasyTL
-
-## custom modules
-from modules.common.toolkit import Toolkit
-from modules.common.file_ensurer import FileEnsurer
-from modules.common.logger import Logger
-from modules.common.decorators import permission_error_decorator
-from modules.common.exceptions import AuthorizationException, QuotaExceededException
-
-##-------------------start-of-Kaiseki--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-
-class Kaiseki:
-
-    """
-
-    Kaiseki is a secondary class that is used to interact with the Deepl API and translate Japanese text sentence by sentence.
-
-    Kaiseki is considered inferior to Kijiku, please consider using Kijiku instead.
-    
-    """
-
-    ##---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-
-    text_to_translate = []
-
-    translated_text = []
-
-    je_check_text = []
-
-    error_text = []
-
-    translation_print_result = ""
-    
-    ##---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-
-    sentence_parts = []
-
-    sentence_punctuation = []
-
-    ## [0] = "" [1] = ~ [2] = '' in Kaiseki.current_sentence but not entire Kaiseki.current_sentence [3] = '' but entire Kaiseki.current_sentence [3] if () in Kaiseki.current_sentence
-    special_punctuation = [] 
-
-    current_sentence = ""
-
-    translated_sentence = ""
-        
-##-------------------start-of-translate()--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-
-    @staticmethod
-    def translate() -> None:
-
-        """
-
-        Translates the text.
-
-        """
-
-        Logger.clear_batch()
-
-        time_start = time.time()
-
-        try:
-
-            Kaiseki.initialize() 
-
-            ## offset time, for if the user doesn't get through Kaiseki.initialize() before the translation starts.
-            time_start = time.time()
-
-            Kaiseki.commence_translation()
-
-        except Exception as e:
-            
-            Kaiseki.translation_print_result += "An error has occurred, outputting results so far..."
-
-            FileEnsurer.handle_critical_exception(e)
-
-        finally:
-
-            time_end = time.time()
-
-            Kaiseki.assemble_results(time_start, time_end)
-
-##-------------------start-of-initialize()--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-
-    @staticmethod
-    def initialize() -> None:
-        
-        """
-
-        Initializes the Kaiseki class by getting the API key and creating the translator object.
-
-        """
-        
-        ## get saved api key if exists
-        try:
-
-            with open(FileEnsurer.deepl_api_key_path, 'r', encoding='utf-8') as file:
-                api_key = base64.b64decode((file.read()).encode('utf-8')).decode('utf-8')
-
-            EasyTL.set_api_key("deepl", api_key)
-            is_valid, e = EasyTL.test_api_key_validity("deepl")
-
-            assert is_valid == True, e
-
-            Logger.log_action("Used saved api key in " + FileEnsurer.deepl_api_key_path, output=True)
-
-        ## else try to get api key manually
-        except Exception as e: 
-
-            api_key = input("DO NOT DELETE YOUR COPY OF THE API KEY\n\nPlease enter the deepL api key you have : ")
-
-            ## if valid save the api key
-            try: 
-
-                EasyTL.set_api_key("deepl", api_key)
-                is_valid, e = EasyTL.test_api_key_validity("deepl")
-
-                assert is_valid, e
-
-                time.sleep(.1)
-                    
-                FileEnsurer.standard_overwrite_file(FileEnsurer.deepl_api_key_path, base64.b64encode(api_key.encode('utf-8')).decode('utf-8'), omit=True)
-
-                time.sleep(.1)
-                
-            ## if invalid key exit
-            except AuthorizationException: 
-                    
-                Toolkit.clear_console()
-                    
-                Logger.log_action("Authorization error with creating translator object, please double check your api key as it appears to be incorrect.\nKaiseki will now exit.", output=True)
-                
-                Toolkit.pause_console()
-                    
-                raise e # type: ignore
-
-            ## other error, alert user and raise it
-            except Exception as e: 
-                
-                Toolkit.clear_console()
-                    
-                Logger.log_action("Unknown error with creating translator object, The error is as follows " + str(e)  + "\nKaiseki will now exit.", output=True)
-                
-                Toolkit.pause_console()
-
-                raise e
-
-        Toolkit.clear_console()
-        Logger.log_barrier()
-
-##-------------------start-of-reset_static_variables()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-
-    @staticmethod
-    def reset_static_variables() -> None:
-
-        """
-
-        Resets the static variables of the Kaiseki class.
-        For when running multiple translations in a row through webgui.
-
-        """
-
-        Logger.clear_batch()
-
-        Kaiseki.text_to_translate = []
-        Kaiseki.translated_text = []
-        Kaiseki.je_check_text = []
-        Kaiseki.error_text = []
-        Kaiseki.translation_print_result = ""
-        Kaiseki.sentence_parts = []
-        Kaiseki.sentence_punctuation = []
-        Kaiseki.special_punctuation = []
-        Kaiseki.current_sentence = ""
-        Kaiseki.translated_sentence = ""
-
-##-------------------start-of-commence_translation()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-
-    @staticmethod
-    def commence_translation() -> None:
-
-        """
-        
-        Commences the translation process using all the functions in the Kaiseki class.
-
-        """
-
-        i = 0 
-
-        while(i < len(Kaiseki.text_to_translate)):
-
-            ## for webgui, if the user presses the clear button, raise an exception to stop the translation
-            if(FileEnsurer.do_interrupt == True):
-                raise Exception("Interrupted by user.")
-
-            Kaiseki.current_sentence = Kaiseki.text_to_translate[i]
-            
-            Logger.log_action("Initial Sentence : " + Kaiseki.current_sentence)
-
-            ## Kaiseki is an in-place translation, so it'll build the translated text into Kaiseki.translated_text as it goes.
-            if(any(char in Kaiseki.current_sentence for char in ["▼", "△", "◇"])):
-                Kaiseki.translated_text.append(Kaiseki.current_sentence + '\n')
-                Logger.log_action("Sentence : " + Kaiseki.current_sentence + ", Sentence is a pov change... leaving intact.")
-
-            elif("part" in Kaiseki.current_sentence.lower() or all(char in ["１","２","３","４","５","６","７","８","９", " "] for char in Kaiseki.current_sentence) and not all(char in [" "] for char in Kaiseki.current_sentence) and Kaiseki.current_sentence != '"..."' and Kaiseki.current_sentence != "..."):
-                Kaiseki.translated_text.append(Kaiseki.current_sentence + '\n') 
-                Logger.log_action("Sentence : " + Kaiseki.current_sentence + ", Sentence is part marker... leaving intact.")
-
-            elif bool(re.match(r'^[\W_\s\n-]+$', Kaiseki.current_sentence)) and not any(char in Kaiseki.current_sentence for char in ["」", "「", "«", "»"]):
-                Logger.log_action("Sentence : " + Kaiseki.current_sentence + ", Sentence is punctuation... skipping.")
-                Kaiseki.translated_text.append(Kaiseki.current_sentence + "\n")
-
-            elif(bool(re.match(r'^[A-Za-z0-9\s\.,\'\?!]+\n*$', Kaiseki.current_sentence))):
-                Logger.log_action("Sentence : " + Kaiseki.current_sentence + ", Sentence is english... skipping translation.")
-                Kaiseki.translated_text.append(Kaiseki.current_sentence + "\n")
-
-            elif(len(Kaiseki.current_sentence) == 0 or Kaiseki.current_sentence.isspace() == True):
-                Logger.log_action("Sentence is empty... skipping translation.\n")
-                Kaiseki.translated_text.append(Kaiseki.current_sentence + "\n") 
-
-            else:
-        
-                Kaiseki.separate_sentence()
-
-                Kaiseki.translate_sentence()
-
-                ## this is for adding a period if it's missing 
-                if(len(Kaiseki.translated_text[i]) > 0 and Kaiseki.translated_text[i] != "" and Kaiseki.translated_text[i][-2] not in string.punctuation and Kaiseki.sentence_punctuation[-1] == None): 
-                    Kaiseki.translated_text[i] = Kaiseki.translated_text[i] + "."
-                
-                ## re-adds quotes
-                if(Kaiseki.special_punctuation[0] == True): 
-                    Kaiseki.translated_text[i] =  '"' + Kaiseki.translated_text[i] + '"'
-
-                ## replaces quotes because deepL messes up quotes
-                elif('"' in Kaiseki.translated_text[i]): 
-                    Kaiseki.translated_text[i] = Kaiseki.translated_text[i].replace('"',"'")
-
-                ## re-adds single quotes
-                if(Kaiseki.special_punctuation[3] == True): 
-                    Kaiseki.translated_text[i] =  "'" + Kaiseki.translated_text[i] + "'"
-
-                ## re-adds parentheses
-                if(Kaiseki.special_punctuation[4] == True): 
-                    Kaiseki.translated_text[i] =  "(" + Kaiseki.translated_text[i] + ")"
-
-                Logger.log_action("Translated and Reassembled Sentence : " + Kaiseki.translated_text[i])
-
-                Kaiseki.translated_text[i] += "\n"
-
-                Kaiseki.je_check_text.append(str(i+1) + ": " + Kaiseki.current_sentence +  "\n   " +  Kaiseki.translated_text[i] + "\n")
-            
-            i+=1
-            
-            Toolkit.clear_console()
-            
-            Logger.log_action(str(i) + "/" + str(len(Kaiseki.text_to_translate)) + " completed.", output=True)
-            Logger.log_barrier()
-
-##-------------------start-of-separate_sentence()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-    
-    @staticmethod
-    def separate_sentence() -> None: 
-
-        """
-
-        This function separates the sentence into parts and punctuation.
-        
-        """
-
-        ## resets variables for current_sentence
-        Kaiseki.sentence_parts = [] 
-        Kaiseki.sentence_punctuation = []
-        Kaiseki.special_punctuation = [False,False,False,False,False] 
-
-        i = 0
-
-        buildString = ""
-
-        ## checks if quotes are in the sentence and removes them
-        if('"' in Kaiseki.current_sentence):
-            Kaiseki.current_sentence = Kaiseki.current_sentence.replace('"', '')
-            Kaiseki.special_punctuation[0] = True
-
-        ## checks if tildes are in the sentence
-        if('~' in Kaiseki.current_sentence):
-            Kaiseki.special_punctuation[1] = True
-
-        ## checks if apostrophes are in the sentence but not at the beginning or end
-        if(Kaiseki.current_sentence.count("'") == 2 and (Kaiseki.current_sentence[0] != "'" and Kaiseki.current_sentence[-1] != "'")):
-            Kaiseki.special_punctuation[2] = True
-
-        ## checks if apostrophes are in the sentence and removes them
-        elif(Kaiseki.current_sentence.count("'") == 2):
-            Kaiseki.special_punctuation[3] = True
-            Kaiseki.current_sentence = Kaiseki.current_sentence.replace("'", "")
-
-        ## checks if parentheses are in the sentence and removes them
-        if("(" in Kaiseki.current_sentence and ")" in Kaiseki.current_sentence):
-            Kaiseki.special_punctuation[4] = True
-            Kaiseki.current_sentence= Kaiseki.current_sentence.replace("(","")
-            Kaiseki.current_sentence= Kaiseki.current_sentence.replace(")","")
-
-        while(i < len(Kaiseki.current_sentence)):
-                
-            if(Kaiseki.current_sentence[i] in [".","!","?","-"]): 
-
-                if(i+5 < len(Kaiseki.current_sentence) and Kaiseki.current_sentence[i:i+6] in ["......"]):
-
-                    if(i+6 < len(Kaiseki.current_sentence) and Kaiseki.current_sentence[i:i+7] in ["......'"]):
-                        buildString += "'"
-                        i+=1
-
-                    if(buildString != ""):
-                        Kaiseki.sentence_parts.append(buildString)
-
-                    Kaiseki.sentence_punctuation.append(Kaiseki.current_sentence[i:i+6])
-                    i+=5
-                    buildString = ""
-                    
-                if(i+4 < len(Kaiseki.current_sentence) and Kaiseki.current_sentence[i:i+5] in [".....","...!?"]):
-
-                    if(i+5 < len(Kaiseki.current_sentence) and Kaiseki.current_sentence[i:i+6] in [".....'","...!?'"]):
-                        buildString += "'"
-                        i+=1
-
-                    if(buildString != ""):
-                        Kaiseki.sentence_parts.append(buildString)
-
-                    Kaiseki.sentence_punctuation.append(Kaiseki.current_sentence[i:i+5])
-                    i+=4
-                    buildString = ""
-                                            
-                elif(i+3 < len(Kaiseki.current_sentence) and Kaiseki.current_sentence[i:i+4] in ["...!","...?","---.","....","!..."]):
-
-                    if(i+4 < len(Kaiseki.current_sentence) and Kaiseki.current_sentence[i:i+5] in ["...!'","...?'","---.'","....'","!...'"]):
-                        buildString += "'"
-                        i+=1
-
-                    if(buildString != ""):
-                        Kaiseki.sentence_parts.append(buildString)
-
-                    Kaiseki.sentence_punctuation.append(Kaiseki.current_sentence[i:i+4])
-                    i+=3
-                    buildString = ""
-                        
-                elif(i+2 < len(Kaiseki.current_sentence) and Kaiseki.current_sentence[i:i+3] in ["---","..."]):
-
-                    if(i+3 < len(Kaiseki.current_sentence) and Kaiseki.current_sentence[i:i+4] in ["---'","...'"]):
-                        buildString += "'"
-                        i+=1
-
-                    if(buildString != ""):
-                        Kaiseki.sentence_parts.append(buildString)
-
-                    Kaiseki.sentence_punctuation.append(Kaiseki.current_sentence[i:i+3])
-                    i+=2
-                    buildString = ""
-
-                elif(i+1 < len(Kaiseki.current_sentence) and Kaiseki.current_sentence[i:i+2] == '!?'):
-
-                    if(i+2 < len(Kaiseki.current_sentence) and Kaiseki.current_sentence[i:i+3] == "!?'"):
-                        buildString += "'"
-                        i+=1
-
-                    if(buildString != ""):
-                        Kaiseki.sentence_parts.append(buildString)
-
-                    Kaiseki.sentence_punctuation.append(Kaiseki.current_sentence[i:i+2])
-                    i+=1
-                    buildString = ""
-                        
-                ## if punctuation that was found is not a hyphen then just follow normal punctuation separation rules
-                elif(Kaiseki.current_sentence[i] != "-"):
-
-                    if(i+1 < len(Kaiseki.current_sentence) and Kaiseki.current_sentence[i+1] == "'"):
-                        buildString += "'"
-
-                    if(buildString != ""):
-                        Kaiseki.sentence_parts.append(buildString)
-
-                    Kaiseki.sentence_punctuation.append(Kaiseki.current_sentence[i])
-                    buildString = ""
-
-                ## if it is just a singular hyphen, do not consider it punctuation as they are used in honorifics
-                else:
-                    buildString += Kaiseki.current_sentence[i]
-            else:
-                buildString += Kaiseki.current_sentence[i] 
-
-            i += 1
-                
-        ## if end of line, add none punctuation which means a period needs to be added later
-        if(buildString): 
-            Kaiseki.sentence_parts.append(buildString)
-            Kaiseki.sentence_punctuation.append(None)
-
-        Logger.log_action("Fragmented Sentence Parts " + str(Kaiseki.sentence_parts))
-        Logger.log_action("Sentence Punctuation " + str(Kaiseki.sentence_punctuation))
-        Logger.log_action("Does Sentence Have Special Punctuation : " + str(Kaiseki.special_punctuation))
-
-        ## strip the sentence parts
-        Kaiseki.sentence_parts = [part.strip() for part in Kaiseki.sentence_parts] 
-
-##-------------------start-of-translate_sentence()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-
-    @staticmethod
-    def translate_sentence() -> None:
-
-        """
-
-        This function translates each part of a sentence.
-
-        """
-
-        i = 0
-        ii = 0
-
-        quote = ""
-        error = ""
-        
-        tilde_active = False
-        single_quote_active = False
-
-        while(i < len(Kaiseki.sentence_parts)):
-
-            ## if tilde is present in part, delete it and set tilde active to true, so we can add it back in a bit
-            if(Kaiseki.special_punctuation[1] == True and "~" in Kaiseki.sentence_parts[i]): 
-                Kaiseki.sentence_parts[i] = Kaiseki.sentence_parts[i].replace("~","")
-                tilde_active = True
-
-            ## a quote is present in the sentence, but not enclosing the sentence, we need to isolate it
-            if(Kaiseki.special_punctuation[2] == True and "'" in Kaiseki.sentence_parts[i] and (Kaiseki.sentence_parts[i][0] != "'" and Kaiseki.sentence_parts[i][-1] != "'")):
-                
-                sentence = Kaiseki.sentence_parts[i]
-                substring_start = sentence.index("'")
-                substring_end = 0
-                quote = ""
-
-                ii = substring_start
-                while(ii < len(sentence)):
-                    if(sentence[ii] == "'"):
-                        substring_end = ii
-                    ii+=1
-                    
-                quote = sentence[substring_start+1:substring_end]
-                Kaiseki.sentence_parts[i] = sentence[:substring_start+1] + "quote" + sentence[substring_end:]
-
-                single_quote_active = True
-                
-            try:
-                results = EasyTL.deepl_translate(text=Kaiseki.sentence_parts[i], source_lang= "JA", target_lang="EN-US")
-
-                assert isinstance(results, str), "ValueError: " + str(results)
-
-                translated_part = results.rstrip(''.join(c for c in string.punctuation if c not in "'\""))
-                translated_part = translated_part.rstrip() 
-
-                ## here we re-add the tilde, (note not always accurate but mostly is)
-                if(tilde_active == True): 
-                    translated_part += "~"
-                    tilde_active = False
-
-                ## translates the quote and re-adds it back to the sentence part
-                if(single_quote_active == True): 
-                    results = EasyTL.deepl_translate(text=Kaiseki.sentence_parts[i], source_lang= "JA", target_lang="EN-US")
-
-                    assert isinstance(results, str), "ValueError: " + str(results)
-                    
-                    quote = quote.rstrip(''.join(c for c in string.punctuation if c not in "'\""))
-                    quote = quote.rstrip() 
-
-                    translated_part = translated_part.replace("'quote'","'" + quote + "'",1)
-
-                ## if punctuation appears first and before any text, add the punctuation and remove it form the list.
-                if(len(Kaiseki.sentence_punctuation) > len(Kaiseki.sentence_parts)): 
-                    Kaiseki.translated_sentence += Kaiseki.sentence_punctuation[0]
-                    Kaiseki.sentence_punctuation.pop(0)
-
-                if(Kaiseki.sentence_punctuation[i] != None):
-                    Kaiseki.translated_sentence += translated_part + Kaiseki.sentence_punctuation[i] 
-                else:
-                    Kaiseki.translated_sentence += translated_part 
-
-                if(i != len(Kaiseki.sentence_punctuation)-1):
-                    Kaiseki.translated_sentence += " "
-                        
-            except QuotaExceededException as e:
-
-                Logger.log_action("DeepL API quota exceeded.", output=True)
-
-                Toolkit.pause_console()
-                
-                raise e
-                    
-            except ValueError as e:
-
-                if(str(e) == "Text must not be empty."):
-                    Kaiseki.translated_sentence += ""
-                else:
-                    Kaiseki.translated_sentence += "ERROR"
-                    error = str(e)
-
-                    Logger.log_action("Error is : " + error)
-                    Kaiseki.error_text.append("Error is : " + error)
-
-            i+=1
-
-        Kaiseki.translated_text.append(Kaiseki.translated_sentence)
-        Kaiseki.translated_sentence = ""
-
-##-------------------start-of-assemble_results()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-    
-    @staticmethod
-    def assemble_results(time_start:float, time_end:float) -> None:
-
-        """
-
-        Prepares the results of the translation for printing.
-
-        Parameters:
-        time_start (float) : the time the translation started.
-        time_end (float) : the time the translation ended.
-
-        """
-        
-        Kaiseki.translation_print_result += "Time Elapsed : " + Toolkit.get_elapsed_time(time_start, time_end)
-
-        Kaiseki.translation_print_result += "\n\nDebug text have been written to : " + FileEnsurer.debug_log_path
-        Kaiseki.translation_print_result += "\nJ->E text have been written to : " + FileEnsurer.je_check_path
-        Kaiseki.translation_print_result += "\nTranslated text has been written to : " + FileEnsurer.translated_text_path
-        Kaiseki.translation_print_result += "\nErrors have been written to : " + FileEnsurer.error_log_path + "\n"
-
-##-------------------start-of-write_kaiseki_results()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-
-    @staticmethod
-    @permission_error_decorator()
-    def write_kaiseki_results() -> None:
-
-        """
-
-        This function is called to write the results of the Kaiseki translation module to the output directory.
-
-        """
-
-        ## ensures the output directory exists, cause it could get moved or fucked with.
-        FileEnsurer.standard_create_directory(FileEnsurer.output_dir)
-
-        with open(FileEnsurer.error_log_path, 'a+', encoding='utf-8') as file:
-            file.writelines(Kaiseki.error_text)
-
-        with open(FileEnsurer.je_check_path, 'w', encoding='utf-8') as file:
-            file.writelines(Kaiseki.je_check_text) 
-
-        with open(FileEnsurer.translated_text_path, 'w', encoding='utf-8') as file:
-            file.writelines(Kaiseki.translated_text)
-
-        ## Instructions to create a copy of the output for archival
-        FileEnsurer.standard_create_directory(FileEnsurer.archive_dir)
-
-        timestamp = Toolkit.get_timestamp(is_archival=True)
-
-        ## pushes the tl debug log to the file without clearing the file
-        Logger.push_batch()
-        Logger.clear_batch()
-
-        list_of_result_tuples = [('kaiseki_translated_text', Kaiseki.translated_text),
-                                 ('kaiseki_je_check_text', Kaiseki.je_check_text),
-                                 ('kaiseki_error_log', Kaiseki.error_text),
-                                 ('debug_log', FileEnsurer.standard_read_file(Logger.log_file_path))]
-
-        FileEnsurer.archive_results(list_of_result_tuples, 
-                                    module='kaiseki', timestamp=timestamp)

modules/common/exceptions.py

@@ -1,12 +1,33 @@
 ## third-party libraries
 ## for importing, other scripts will use from common.exceptions instead of from the third-party libraries themselves
-from easytl import AuthenticationError, InternalServerError, RateLimitError, APITimeoutError
-from easytl import AuthorizationException, QuotaExceededException
-from easytl import GoogleAuthError
+from easytl import OpenAIAuthenticationError, OpenAIInternalServerError, OpenAIRateLimitError, OpenAIAPITimeoutError, OpenAIAPIConnectionError, OpenAIAPIStatusError
+from easytl import DeepLAuthorizationException, DeepLQuotaExceededException, DeepLException
+from easytl import GoogleAuthError, GoogleAPIError
+
+##-------------------start-of-KudasaiException--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+
+class KudasaiException(Exception):
+
+    """
+
+    KudasaiException is an exception that is raised when an error occurs within the Kudasai Application.
+
+    """
+
+    def __init__(self, message:str) -> None:
+
+        """
+
+        Parameters:
+        message (string) : The message to display.
+
+        """
+
+        self.message = message
 
 ##-------------------start-of-MaxBatchDurationExceededException--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-class MaxBatchDurationExceededException(Exception):
+class MaxBatchDurationExceededException(KudasaiException):
 
     """
 
@@ -27,7 +48,7 @@ class MaxBatchDurationExceededException(Exception):
 
 ##-------------------start-of-InvalidAPIKeyException--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-class InvalidAPIKeyException(Exception):
+class InvalidAPIKeyException(KudasaiException):
 
     """
 
@@ -48,7 +69,7 @@ class InvalidAPIKeyException(Exception):
 
 ##-------------------start-of-TooManyFileAccessAttemptsException--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-class TooManyFileAccessAttemptsException(Exception):
+class TooManyFileAccessAttemptsException(KudasaiException):
 
     """
 

modules/common/file_ensurer.py

@@ -4,10 +4,10 @@ import traceback
 import json
 import typing
 import shutil
+import logging
 
 ## custom modules
 from modules.common.decorators import permission_error_decorator
-from modules.common.logger import Logger
 from modules.common.toolkit import Toolkit
 
 class FileEnsurer():
@@ -16,7 +16,7 @@ class FileEnsurer():
     
     FileEnsurer is a class that is used to ensure that the required files and directories exist.
     Also serves as a place to store the paths to the files and directories. Some file related functions are also stored here.
-    As well as some variables that are used to store the default kijiku rules and the allowed models across Kudasai.
+    As well as some variables that are used to store the default translation settings and the allowed models across Kudasai.
 
     """
 
@@ -28,37 +28,36 @@ class FileEnsurer():
     hugging_face_flag = os.path.join(script_dir, "util", "hugging_face_flag.py")
 
     ## main dirs (config is just under userprofile on windows, and under home on linux); secrets are under appdata on windows, and under .config on linux
-    if(os.name == 'nt'):  ## Windows
+    if(Toolkit.is_windows()):  ## Windows
         config_dir = os.path.join(os.environ['USERPROFILE'],"KudasaiConfig")
         secrets_dir = os.path.join(os.environ['APPDATA'],"KudasaiSecrets")
     else:  ## Linux
         config_dir = os.path.join(os.path.expanduser("~"), "KudasaiConfig")
         secrets_dir = os.path.join(os.path.expanduser("~"), ".config", "KudasaiSecrets")
 
-    Logger.log_file_path = os.path.join(output_dir, "debug_log.txt")
-
     ##----------------------------------/
 
     ## sub dirs
     lib_dir = os.path.join(script_dir, "lib")
+    common_lib = os.path.join(lib_dir, "common")
     gui_lib = os.path.join(lib_dir, "gui")
     jsons_dir = os.path.join(script_dir, "jsons")
 
     ##----------------------------------/
 
     ## output files
-    preprocessed_text_path = os.path.join(output_dir, "preprocessed_text.txt") ## path for the preprocessed text
-    translated_text_path = os.path.join(output_dir, "translated_text.txt") ## path for translated text
+    preprocessed_text_path = os.path.join(output_dir, "preprocessed_text.txt") 
+    translated_text_path = os.path.join(output_dir, "translated_text.txt")
 
-    je_check_path = os.path.join(output_dir, "je_check_text.txt") ## path for je check text (text generated by the translation modules to compare against the translated text)
+    je_check_path = os.path.join(output_dir, "je_check_text.txt") 
 
-    kairyou_log_path = os.path.join(output_dir, "preprocessing_results.txt")  ## path for kairyou log (the results of preprocessing)
-    error_log_path = os.path.join(output_dir, "error_log.txt") ## path for the error log (errors generated by the preprocessing and translation modules)
-    debug_log_path = Logger.log_file_path ## path for the debug log (debug info generated by the preprocessing and translation modules)
+    kairyou_log_path = os.path.join(output_dir, "preprocessing_results.txt")  
+    error_log_path = os.path.join(output_dir, "error_log.txt") 
+    debug_log_path = os.path.join(output_dir, "debug_log.txt") 
  
-    ## kijiku rules
-    external_kijiku_rules_path = os.path.join(script_dir,'kijiku_rules.json')
-    config_kijiku_rules_path = os.path.join(config_dir,'kijiku_rules.json')
+    ## translation settings
+    external_translation_settings_path = os.path.join(script_dir,'translation_settings.json')
+    config_translation_settings_path = os.path.join(config_dir,'translation_settings.json')
 
     ## api keys
     deepl_api_key_path = os.path.join(secrets_dir, "deepl_api_key.txt")
@@ -68,8 +67,14 @@ class FileEnsurer():
     ## favicon
     favicon_path = os.path.join(gui_lib, "Kudasai_Logo.png")
 
-    DEFAULT_KIJIKU_RULES = {
-    "base kijiku settings": {
+    ## js save to file
+    js_save_to_file_path = os.path.join(gui_lib, "save_to_file.js")
+
+    ## translation settings description
+    translation_settings_description_path = os.path.join(common_lib, "translation_settings_description.txt")
+
+    DEFAULT_TRANSLATION_SETTING = {
+    "base translation settings": {
         "prompt_assembly_mode": 1,
         "number_of_lines_per_batch": 36,
         "sentence_fragmenter_mode": 2,
@@ -103,9 +108,16 @@ class FileEnsurer():
         "gemini_stream": False,
         "gemini_stop_sequences": None,
         "gemini_max_output_tokens": None
+    },
+
+    "deepl settings":{
+        "deepl_context": "",
+        "deepl_split_sentences": "ALL",
+        "deepl_preserve_formatting": True,
+        "deepl_formality": "default"
     }
 }
-    INVALID_KIJIKU_RULES_PLACEHOLDER = {
+    INVALID_TRANSLATION_SETTINGS_PLACEHOLDER = {
     "INVALID JSON": 
     {
         "INVALID JSON":"INVALID JSON"
@@ -127,6 +139,9 @@ class FileEnsurer():
         
         Determines if Kudasai is running on a Hugging Face server.
 
+        Returns:
+        bool : whether or not Kudasai is running on a Hugging Face server.
+
         """
 
         return os.path.exists(FileEnsurer.hugging_face_flag)
@@ -144,8 +159,6 @@ class FileEnsurer():
 
         print("Cleaning up and exiting...")
 
-        Logger.push_batch()
-
         exit()
 
 ##-------------------start-of-setup_needed_files()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
@@ -158,6 +171,9 @@ class FileEnsurer():
 
         Ensures that the required files and directories exist.
 
+        Decorated By:
+        permission_error_decorator
+
         """
 
 
@@ -168,9 +184,6 @@ class FileEnsurer():
         FileEnsurer.standard_create_directory(FileEnsurer.output_dir)
         FileEnsurer.standard_create_directory(FileEnsurer.secrets_dir)
 
-        ## creates and clears the log file
-        Logger.clear_log_file()
-
         ## creates the 5 output files
         FileEnsurer.standard_create_file(FileEnsurer.preprocessed_text_path)
         FileEnsurer.standard_create_file(FileEnsurer.translated_text_path)
@@ -179,9 +192,9 @@ class FileEnsurer():
         FileEnsurer.standard_create_file(FileEnsurer.error_log_path)
 
         ## creates the kijiku rules file if it doesn't exist
-        if(os.path.exists(FileEnsurer.config_kijiku_rules_path) == False):
-            with open(FileEnsurer.config_kijiku_rules_path, 'w+', encoding='utf-8') as file:
-                json.dump(FileEnsurer.DEFAULT_KIJIKU_RULES, file)
+        if(os.path.exists(FileEnsurer.config_translation_settings_path) == False):
+            with open(FileEnsurer.config_translation_settings_path, 'w+', encoding='utf-8') as file:
+                json.dump(FileEnsurer.DEFAULT_TRANSLATION_SETTING, file)
 
 ##-------------------start-of-purge_storage()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
                 
@@ -193,10 +206,16 @@ class FileEnsurer():
         
         In case of hugging face, purges the storage.
 
+        Decorated By:
+        permission_error_decorator
+
         """
 
         if(not FileEnsurer.is_hugging_space()):
+            logging.debug("Not running on Hugging Face, skipping storage purge")
             return
+        
+        logging.debug("Running on Hugging Face, purging storage")
 
         stuff_to_purge = [
             FileEnsurer.secrets_dir,
@@ -243,11 +262,14 @@ class FileEnsurer():
         Parameters:
         directory_path (str) : path to the directory to be created.
 
+        Decorated By:
+        permission_error_decorator
+
         """
 
         if(os.path.isdir(directory_path) == False):
             os.makedirs(directory_path)
-            Logger.log_action(directory_path + " created due to lack of the folder")
+            logging.debug(directory_path + " created due to lack of the folder")
 
 ##--------------------start-of-standard_create_file()------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
@@ -262,10 +284,13 @@ class FileEnsurer():
         Parameters:
         file_path (str) : path to the file to be created.
 
+        Decorated By:
+        permission_error_decorator
+
         """
 
         if(os.path.exists(file_path) == False):
-            Logger.log_action(file_path + " was created due to lack of the file")
+            logging.debug(file_path + " was created due to lack of the file")
             with open(file_path, "w+", encoding="utf-8") as file:
                 file.truncate()
 
@@ -286,12 +311,15 @@ class FileEnsurer():
         Returns:
         bool : whether or not the file was overwritten.
 
+        Decorated By:
+        permission_error_decorator
+
         """
 
         did_overwrite = False
 
         if(os.path.exists(file_path) == False or os.path.getsize(file_path) == 0):
-            Logger.log_action(file_path + " was created due to lack of the file or because it is blank")
+            logging.debug(file_path + " was created due to lack of the file or because it is blank")
             with open(file_path, "w+", encoding="utf-8") as file:
                 file.write(content_to_write)
 
@@ -312,7 +340,10 @@ class FileEnsurer():
         Parameters:
         file_path (str) : path to the file to be overwritten.
         content to write (str) : content to be written to the file.
-        omit (bool | optional) : whether or not to omit the content from the log.
+        omit (bool | optional | default=True) : whether or not to omit the content from the log.
+
+        Decorated By:
+        permission_error_decorator
 
         """
 
@@ -322,7 +353,7 @@ class FileEnsurer():
         if(omit):
             content_to_write = "(Content was omitted)"
         
-        Logger.log_action(file_path + " was overwritten with the following content: " + content_to_write)
+        logging.debug(file_path + " was overwritten with the following content: " + content_to_write)
 
 ##--------------------start-of-clear_file()------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
@@ -337,12 +368,15 @@ class FileEnsurer():
         Parameters:
         file_path (str) : path to the file to be cleared.
 
+        Decorated By:
+        permission_error_decorator
+
         """
 
         with open(file_path, "w+", encoding="utf-8") as file:
             file.truncate()
 
-        Logger.log_action(file_path + " was cleared")
+        logging.debug(file_path + " was cleared")
 
 ##--------------------start-of-standard_read_file()------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
@@ -360,6 +394,9 @@ class FileEnsurer():
         Returns:
         content (str) : the content of the file.
 
+        Decorated By:
+        permission_error_decorator
+
         """
 
         with open(file_path, "r", encoding="utf-8") as file:
@@ -379,17 +416,16 @@ class FileEnsurer():
         Parameters:
         critical_exception (object - Exception) : the exception to be handled.
 
-        """
+        Decorated By:
+        permission_error_decorator
 
-        Logger.log_barrier()
-        Logger.log_action("Kudasai has crashed", output=True)
-        Logger.log_action("Please send the following to the developer on github at https://github.com/Bikatr7/Kudasai/issues :", output=True, omit_timestamp=True)
+        """
 
         traceback_msg = traceback.format_exc()
 
-        Logger.log_action(traceback_msg ,output=True, omit_timestamp=True)
-
-        Logger.push_batch()
+        logging.error(f"Kudasai has crashed "
+        f"Please send the following to the developer on github at https://github.com/Bikatr7/Kudasai/issues :"
+        f"{traceback_msg}")
 
         Toolkit.pause_console()
 
@@ -411,6 +447,9 @@ class FileEnsurer():
         module (str) : name of the module that generated the results.
         timestamp (str) : timestamp of when the results were generated.
 
+        Decorated By:
+        permission_error_decorator
+
         """
 
         archival_path = os.path.join(FileEnsurer.archive_dir, f'{module}_run_{timestamp}')
@@ -439,6 +478,9 @@ class FileEnsurer():
         Returns:
         json_object (dict) : the json object.
 
+        Decorated By:
+        permission_error_decorator
+
         """
 
         with open(file_path, "r", encoding="utf-8") as file:
@@ -462,6 +504,9 @@ class FileEnsurer():
         error_log (str) : the log of any errors that occurred during preprocessing.
         timestamp (str) : the timestamp of when the results were generated (Can be obtained from Toolkit.get_timestamp(is_archival=True))
 
+        Decorated By:
+        permission_error_decorator
+
         """
 
         ## ensures the output directory exists, cause it could get moved or fucked with.
@@ -485,14 +530,12 @@ class FileEnsurer():
         ## Instructions to create a copy of the output for archival
         FileEnsurer.standard_create_directory(FileEnsurer.archive_dir)
 
-        Logger.push_batch()
-        Logger.clear_batch()
-
         list_of_result_tuples = [('kairyou_preprocessed_text', text_to_preprocess),
-                                 ('kairyou_preprocessing_log', preprocessing_log),
-                                 ('kairyou_error_log', error_log),
-                                 ('debug_log', FileEnsurer.standard_read_file(Logger.log_file_path))]
+                     ('kairyou_preprocessing_log', preprocessing_log),
+                     ('kairyou_error_log', error_log),
+                     ('debug_log', FileEnsurer.standard_read_file(FileEnsurer.debug_log_path))]
 
         FileEnsurer.archive_results(list_of_result_tuples,
-                                    module='kairyou', timestamp=timestamp)
+                        module='kairyou', 
+                        timestamp=timestamp)
 

modules/common/logger.py

@@ -1,132 +0,0 @@
-## custom modules
-from modules.common.toolkit import Toolkit
-from modules.common.decorators import permission_error_decorator
-
-class Logger:
-
-    """
-    
-    The logger class is used to log actions taken by Kudasai.
-
-    """
-
-    log_file_path = ""
-
-    current_batch = ""
-
-    errors = []
-    
-##--------------------start-of-log_action()------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-
-    @staticmethod
-    def log_action(action:str, output:bool=False, omit_timestamp:bool=False) -> None:
-
-        """
-        
-        Logs an action.
-
-        Parameters:
-        action (str) : the action being logged.
-        output (bool | optional | defaults to false) : whether or not to output the action to the console.
-        omit_timestamp (bool | optional | defaults to false) : whether or not to omit the timestamp from the action.
- 
-        """
-
-        timestamp = Toolkit.get_timestamp() 
-
-        log_line = timestamp + action + "\n"
-
-        Logger.current_batch += log_line
-
-        if(omit_timestamp):
-            log_line = action
-
-        if(output):
-            print(log_line)
-
-##--------------------start-of-log_error()------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-    
-    @staticmethod
-    def log_error(action:str, output:bool=False, omit_timestamp:bool=False) -> None:
-
-        """
-        
-        Logs an error.
-
-        Parameters:
-        action (str) : the action being logged.
-        output (bool | optional | defaults to false) : whether or not to output the action to the console.
-        omit_timestamp (bool | optional | defaults to false) : whether or not to omit the timestamp from the action.
- 
-        """
-
-        timestamp = Toolkit.get_timestamp() 
-
-        log_line = timestamp + action + "\n"
-
-        Logger.current_batch += log_line
-
-        if(omit_timestamp):
-            log_line = action
-
-        if(output):
-            print(log_line)
-
-        Logger.errors.append(log_line)
-
-##--------------------start-of-log_barrier()------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-
-    @staticmethod
-    def log_barrier() -> None:
-            
-        """
-        
-        Logs a barrier.
-
-        """
-    
-        Logger.log_action("-------------------------")
-
-##--------------------start-of-clear_batch()------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-
-    @staticmethod
-    def clear_batch() -> None:
-
-        """
-        
-        Clears the current batch.
-
-        """
-
-        Logger.current_batch = ""
-        Logger.errors = []
-
-##--------------------start-of-push_batch()------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-
-    @staticmethod
-    @permission_error_decorator()
-    def push_batch() -> None:
-
-        """
-        
-        Pushes all stored actions to the log file.
-
-        """
-
-        with open(Logger.log_file_path, 'a+', encoding="utf-8") as file:
-            file.write(Logger.current_batch)
-            
-##--------------------start-of-clear_log_file()------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-
-    @staticmethod
-    @permission_error_decorator()
-    def clear_log_file() -> None:
-
-        """
-        
-        Clears the log file.
-        
-        """
-
-        with open(Logger.log_file_path, 'w+', encoding="utf-8") as file:
-            file.truncate(0)

modules/common/toolkit.py

@@ -3,6 +3,7 @@ from datetime import datetime
 
 import os
 import typing
+import logging
 import platform
 import subprocess
 
@@ -14,7 +15,7 @@ class Toolkit():
 
     """
 
-    CURRENT_VERSION = "v3.4.3"
+    CURRENT_VERSION = "v3.4.5"
 
 ##-------------------start-of-clear_console()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
@@ -29,6 +30,22 @@ class Toolkit():
 
         os.system('cls' if os.name == 'nt' else 'clear')
 
+##-------------------start-of-is_windows()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+
+    @staticmethod
+    def is_windows() -> bool:
+
+        """
+        
+        Returns True if Kudasai is running on Windows.
+
+        Returns:
+        is_windows (bool) : If Kudasai is running on Windows.
+
+        """
+
+        return os.name == 'nt'
+
 ##-------------------start-of-pause_console()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
     @staticmethod
@@ -74,7 +91,6 @@ class Toolkit():
                     termios.tcsetattr(0, termios.TCSANOW, old_settings)
 
         except ImportError:
-
             pass
 
 ##-------------------start-of-maximize_window()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
@@ -164,12 +180,15 @@ class Toolkit():
 ##-------------------start-of-check_update()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
     @staticmethod
-    def check_update() -> typing.Tuple[bool, str]:
+    def check_update(do_pause:bool=True) -> typing.Tuple[bool, str]:
 
         """
 
         Determines if Kudasai has a new latest release, and confirms if an internet connection is present or not.
 
+        Parameters:
+        do_pause (bool | optional | default=True) : Whether or not to pause the console after displaying the update prompt.
+
         Returns:
         is_connection (bool) : Whether or not the user has an internet connection.
         update_prompt (str) : The update prompt to be displayed to the user, can either be blank if there is no update or contain the update prompt if there is an update.
@@ -193,6 +212,8 @@ class Toolkit():
 
             if(LooseVersion(latest_version) > LooseVersion(Toolkit.CURRENT_VERSION)):
 
+                logging.debug("New update available: " + latest_version)
+
                 update_prompt += "There is a new update for Kudasai (" + latest_version + ")\nIt is recommended that you use the latest version of Kudasai\nYou can download it at https://github.com/Bikatr7/Kudasai/releases/latest \n"
 
                 if(release_notes):
@@ -203,9 +224,12 @@ class Toolkit():
         ## used to determine if user lacks an internet connection.
         except:
 
+            logging.debug("No internet connection detected.")
+
             print("You seem to lack an internet connection, this will prevent you from checking from update notification and machine translation.\n")
 
-            Toolkit.pause_console()
+            if(do_pause):
+                Toolkit.pause_console()
 
             is_connection = False
 

models/kijiku.py → modules/common/translator.py

@@ -6,6 +6,7 @@ import time
 import typing
 import asyncio
 import os
+import logging
 
 ## third party modules
 from kairyou import KatakanaUtil
@@ -17,19 +18,18 @@ import backoff
 from handlers.json_handler import JsonHandler
 
 from modules.common.file_ensurer import FileEnsurer
-from modules.common.logger import Logger
 from modules.common.toolkit import Toolkit
-from modules.common.exceptions import AuthenticationError, MaxBatchDurationExceededException, AuthenticationError, InternalServerError, RateLimitError, APITimeoutError, GoogleAuthError
+from modules.common.exceptions import OpenAIAuthenticationError, MaxBatchDurationExceededException, DeepLAuthorizationException, OpenAIInternalServerError, OpenAIRateLimitError, OpenAIAPITimeoutError, GoogleAuthError, OpenAIAPIStatusError, OpenAIAPIConnectionError, DeepLException, GoogleAPIError
 from modules.common.decorators import permission_error_decorator
 
-##-------------------start-of-Kijiku--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+##-------------------start-of-Translator--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-class Kijiku:
+class Translator:
 
     """
     
-    Kijiku is a secondary class that is used to interact with LLMs and translate text.
-    Currently supports OpenAI and Gemini.
+    Translator is a class that is used to interact with translation methods and translate text.
+    Currently supports OpenAI, Gemini, and DeepL.
     
     """
     
@@ -48,6 +48,9 @@ class Kijiku:
     ## meanwhile for gemini, we just need to send the prompt and the text to be translated concatenated together
     gemini_translation_batches:typing.List[str] = []
 
+    ## same as above, but for deepl, just the text to be translated
+    deepl_translation_batches:typing.List[str] = []
+
     num_occurred_malformed_batches = 0
 
     ## semaphore to limit the number of concurrent batches
@@ -55,7 +58,7 @@ class Kijiku:
 
     ##--------------------------------------------------------------------------------------------------------------------------
 
-    LLM_TYPE:typing.Literal["openai", "gemini"] = "openai"
+    TRANSLATION_METHOD:typing.Literal["openai", "gemini", "deepl"] = "openai"
 
     translation_print_result = ""
 
@@ -71,6 +74,10 @@ class Kijiku:
 
     decorator_to_use:typing.Callable
 
+    is_cli = False
+
+    pre_provided_api_key = ""
+
 ##-------------------start-of-get_max_batch_duration()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
     
     @staticmethod
@@ -79,14 +86,14 @@ class Kijiku:
         """
         
         Returns the max batch duration.
-        Structured as a function so that it can be used as a lambda function in the backoff decorator. As decorators call the function when they are defined/runtime, not when they are called.
+        Structured as a function so that it can be used as a lambda function in the backoff decorator. As decorators call the function when they are defined/runtime, not when they are called. Which I learned the hard way.
 
         Returns:
         max_batch_duration (float) : the max batch duration.
 
         """
 
-        return Kijiku.max_batch_duration
+        return Translator.max_batch_duration
     
 ##-------------------start-of-log_retry()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
@@ -104,9 +111,7 @@ class Kijiku:
 
         retry_msg = f"Retrying translation after {details['wait']} seconds after {details['tries']} tries {details['target']} due to {details['exception']}."
 
-        Logger.log_barrier()
-        Logger.log_action(retry_msg)
-        Logger.log_barrier()
+        logging.warning(retry_msg)
 
 ##-------------------start-of-log_failure()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
@@ -120,13 +125,14 @@ class Kijiku:
         Parameters:
         details (dict) : the details of the failure.
 
+        Raises:
+        MaxBatchDurationExceededException : An exception that is raised when the max batch duration is exceeded.
+
         """
 
-        error_msg = f"Exceeded duration, returning untranslated text after {details['tries']} tries {details['target']}."
+        error_msg = f"Exceeded allowed duration of {details['wait']} seconds, returning untranslated text after {details['tries']} tries {details['target']}."
 
-        Logger.log_barrier()
-        Logger.log_error(error_msg)
-        Logger.log_barrier()
+        logging.error(error_msg)
 
         raise MaxBatchDurationExceededException(error_msg)
 
@@ -141,27 +147,26 @@ class Kijiku:
 
         """
 
-        Logger.clear_batch()
-
         ## set this here cause the try-except could throw before we get past the settings configuration
         time_start = time.time()
 
         try:
         
-            await Kijiku.initialize()
+            await Translator.initialize()
 
             JsonHandler.validate_json()
 
-            await Kijiku.check_settings()
+            if(not Translator.is_cli):
+                await Translator.check_settings()
 
             ## set actual start time to the end of the settings configuration
             time_start = time.time()
 
-            await Kijiku.commence_translation()
+            await Translator.commence_translation()
 
         except Exception as e:
             
-            Kijiku.translation_print_result += "An error has occurred, outputting results so far..."
+            Translator.translation_print_result += "An error has occurred, outputting results so far..."
 
             FileEnsurer.handle_critical_exception(e)
 
@@ -169,7 +174,10 @@ class Kijiku:
 
             time_end = time.time() 
 
-            Kijiku.assemble_results(time_start, time_end)
+            Translator.assemble_results(time_start, time_end)
+
+            if(Translator.is_cli):
+                Toolkit.pause_console()
 
 ##-------------------start-of-initialize()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
@@ -178,38 +186,47 @@ class Kijiku:
 
         """
 
-        Sets the API Key for the respective service and loads the kijiku rules.
+        Sets the API Key for the respective service and loads the translation settings.
     
         """
 
-        print("What LLM do you want to use? (1 for OpenAI or 2 for Gemini) : ")
+        translation_methods = {
+            "1": ("openai", FileEnsurer.openai_api_key_path),
+            "2": ("gemini", FileEnsurer.gemini_api_key_path),
+            "3": ("deepl", FileEnsurer.deepl_api_key_path),
+        }
 
-        if(input("\n") == "1"):
-            Kijiku.LLM_TYPE = "openai"
-        
-        else:
-            Kijiku.LLM_TYPE = "gemini"
+        if(not Translator.is_cli):
+            method = input("What method would you like to use for translation? (1 for OpenAI, 2 for Gemini, 3 for Deepl, or any other key to exit) : \n")
 
-        Toolkit.clear_console()
-
-        if(Kijiku.LLM_TYPE == "openai"):
-            await Kijiku.init_api_key("OpenAI", FileEnsurer.openai_api_key_path, EasyTL.set_api_key, EasyTL.test_api_key_validity)
+            if(method not in translation_methods.keys()):
+                print("\nThank you for using Kudasai, goodbye.")
+                time.sleep(2)
+                FileEnsurer.exit_kudasai()
+            
+            Toolkit.clear_console()
 
         else:
-            await Kijiku.init_api_key("Gemini", FileEnsurer.gemini_api_key_path, EasyTL.set_api_key, EasyTL.test_api_key_validity)
-
-        ## try to load the kijiku rules
-        try: 
+            method = Translator.TRANSLATION_METHOD
 
-            JsonHandler.load_kijiku_rules()
+        Translator.TRANSLATION_METHOD, api_key_path = translation_methods.get(method, ("deepl", FileEnsurer.deepl_api_key_path))
+        
+        if(Translator.pre_provided_api_key != ""):
+            encoded_key = base64.b64encode(Translator.pre_provided_api_key.encode('utf-8')).decode('utf-8')
+            Translator.pre_provided_api_key = ""
+            with open(api_key_path, 'w+', encoding='utf-8') as file: 
+                file.write(encoded_key)
 
-        ## if the kijiku rules don't exist, create them
+        await Translator.init_api_key(Translator.TRANSLATION_METHOD.capitalize(), api_key_path, EasyTL.set_credentials, EasyTL.test_credentials)
+        
+        ## try to load the translation settings
+        try: 
+            JsonHandler.load_translation_settings()
+        ## if the translation settings don't exist, create them
         except: 
-            
-            JsonHandler.reset_kijiku_rules_to_default()
-
-            JsonHandler.load_kijiku_rules()
-            
+            JsonHandler.reset_translation_settings_to_default()
+            JsonHandler.load_translation_settings()
+        
         Toolkit.clear_console()
 
 ##-------------------start-of-init_openai_api_key()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
@@ -241,9 +258,8 @@ class Kijiku:
             ## if not valid, raise the exception that caused the test to fail
             if(not is_valid and e is not None):
                 raise e
-        
-            Logger.log_action("Used saved API key in " + api_key_path, output=True)
-            Logger.log_barrier()
+
+            logging.info(f"Used saved API key in {api_key_path}")        
 
             time.sleep(2)
 
@@ -267,22 +283,22 @@ class Kijiku:
                 FileEnsurer.standard_overwrite_file(api_key_path, base64.b64encode(api_key.encode('utf-8')).decode('utf-8'), omit=True)
                 
             ## if invalid key exit
-            except (GoogleAuthError, AuthenticationError):
+            except (GoogleAuthError, OpenAIAuthenticationError, DeepLAuthorizationException):
                     
                 Toolkit.clear_console()
-                        
-                Logger.log_action(f"Authorization error while setting up {service}, please double check your API key as it appears to be incorrect.", output=True)
+
+                logging.error(f"Authorization error while setting up {service}, please double check your API key as it appears to be incorrect.") 
 
                 Toolkit.pause_console()
                         
-                exit()
+                exit(1)
 
             ## other error, alert user and raise it
             except Exception as e: 
 
                 Toolkit.clear_console()
                         
-                Logger.log_action(f"Unknown error while setting up {service}, The error is as follows " + str(e)  + "\nThe exception will now be raised.", output=True)
+                logging.error(f"Unknown error while setting up {service}, The error is as follows " + str(e)  + "\nThe exception will now be raised.")
 
                 Toolkit.pause_console()
 
@@ -300,17 +316,17 @@ class Kijiku:
 
         """
 
-        Logger.clear_batch()
-
-        Kijiku.text_to_translate = []
-        Kijiku.translated_text = []
-        Kijiku.je_check_text = []
-        Kijiku.error_text = []
-        Kijiku.openai_translation_batches = []
-        Kijiku.gemini_translation_batches = []
-        Kijiku.num_occurred_malformed_batches = 0
-        Kijiku.translation_print_result = ""
-        Kijiku.LLM_TYPE = "openai"
+        Translator.text_to_translate = []
+        Translator.translated_text = []
+        Translator.je_check_text = []
+        Translator.error_text = []
+        Translator.openai_translation_batches = []
+        Translator.gemini_translation_batches = []
+        Translator.num_occurred_malformed_batches = 0
+        Translator.translation_print_result = ""
+        Translator.TRANSLATION_METHOD = "openai"
+        Translator.pre_provided_api_key = ""
+        Translator.is_cli = False
 
 ##-------------------start-of-check-settings()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
@@ -319,54 +335,48 @@ class Kijiku:
 
         """
 
-        Prompts the user to confirm the settings in the kijiku rules file.
+        Prompts the user to confirm the settings in the translation settings file.
 
         """
 
         print("Are these settings okay? (1 for yes or 2 for no) : \n\n")
 
+        method_to_section_dict = {
+            "openai": ("openai settings", "OpenAI", FileEnsurer.openai_api_key_path),
+            "gemini": ("gemini settings", "Gemini", FileEnsurer.gemini_api_key_path),
+            "deepl": ("deepl settings", "DeepL", FileEnsurer.deepl_api_key_path)
+        }
+
+        section_to_target, method_name, api_key_path = method_to_section_dict[Translator.TRANSLATION_METHOD]
+
         try:
 
-            JsonHandler.print_kijiku_rules(output=True)
+            JsonHandler.log_translation_settings(output_to_console=True, specific_section=section_to_target)
 
         except:
             Toolkit.clear_console()
 
-            if(input("It's likely that you're using an outdated version of the kijiku rules file, press 1 to reset these to default or 2 to exit and resolve manually : ") == "1"):
+            if(input("It's likely that you're using an outdated version of the translation settings file, press 1 to reset these to default or 2 to exit and resolve manually : ") == "1"):
                 Toolkit.clear_console()
-                JsonHandler.reset_kijiku_rules_to_default()
-                JsonHandler.load_kijiku_rules()
+                JsonHandler.reset_translation_settings_to_default()
+                JsonHandler.load_translation_settings()
 
                 print("Are these settings okay? (1 for yes or 2 for no) : \n\n")
-                JsonHandler.print_kijiku_rules(output=True)
-
+                JsonHandler.log_translation_settings(output_to_console=True, specific_section=section_to_target)
             else:
                 FileEnsurer.exit_kudasai()
 
-        if(input("\n") == "1"):
-            pass
-        else:
-            JsonHandler.change_kijiku_settings()
+        if(input("\n") != "1"):
+            JsonHandler.change_translation_settings()
 
         Toolkit.clear_console()
 
         print("Do you want to change your API key? (1 for yes or 2 for no) : ")
 
         if(input("\n") == "1"):
-
-            if(Kijiku.LLM_TYPE == "openai"):
-
-                if(os.path.exists(FileEnsurer.openai_api_key_path)):
-
-                    os.remove(FileEnsurer.openai_api_key_path)
-                    await Kijiku.init_api_key("OpenAI", FileEnsurer.openai_api_key_path, EasyTL.set_api_key, EasyTL.test_api_key_validity)
-
-            else:
-    
-                if(os.path.exists(FileEnsurer.gemini_api_key_path)):
-
-                    os.remove(FileEnsurer.gemini_api_key_path)
-                    await Kijiku.init_api_key("Gemini", FileEnsurer.gemini_api_key_path, EasyTL.set_api_key, EasyTL.test_api_key_validity)
+            if(os.path.exists(api_key_path)):
+                os.remove(api_key_path)
+                await Translator.init_api_key(method_name, api_key_path, EasyTL.set_credentials, EasyTL.test_credentials)
 
         Toolkit.clear_console()
 
@@ -383,106 +393,106 @@ class Kijiku:
         is_webgui (bool | optional | default=False) : A bool representing whether the function is being called by the webgui.
 
         """
+
+        logging.debug(f"Translator Activated, Translation Method : {Translator.TRANSLATION_METHOD} "
+                     f"Settings are as follows : ")
         
+        JsonHandler.log_translation_settings()
+
+        Translator.prompt_assembly_mode = int(JsonHandler.current_translation_settings["base translation settings"]["prompt_assembly_mode"])
+        Translator.number_of_lines_per_batch = int(JsonHandler.current_translation_settings["base translation settings"]["number_of_lines_per_batch"])
+        Translator.sentence_fragmenter_mode = int(JsonHandler.current_translation_settings["base translation settings"]["sentence_fragmenter_mode"])
+        Translator.je_check_mode = int(JsonHandler.current_translation_settings["base translation settings"]["je_check_mode"])
+        Translator.num_of_malform_retries = int(JsonHandler.current_translation_settings["base translation settings"]["number_of_malformed_batch_retries"])
+        Translator.max_batch_duration = float(JsonHandler.current_translation_settings["base translation settings"]["batch_retry_timeout"])
+        Translator.num_concurrent_batches = int(JsonHandler.current_translation_settings["base translation settings"]["number_of_concurrent_batches"])
+
+        Translator._semaphore = asyncio.Semaphore(Translator.num_concurrent_batches)
+
+        Translator.openai_model = JsonHandler.current_translation_settings["openai settings"]["openai_model"]
+        Translator.openai_system_message = JsonHandler.current_translation_settings["openai settings"]["openai_system_message"]
+        Translator.openai_temperature = float(JsonHandler.current_translation_settings["openai settings"]["openai_temperature"])
+        Translator.openai_top_p = float(JsonHandler.current_translation_settings["openai settings"]["openai_top_p"])
+        Translator.openai_n = int(JsonHandler.current_translation_settings["openai settings"]["openai_n"])
+        Translator.openai_stream = bool(JsonHandler.current_translation_settings["openai settings"]["openai_stream"])
+        Translator.openai_stop = JsonHandler.current_translation_settings["openai settings"]["openai_stop"]
+        Translator.openai_logit_bias = JsonHandler.current_translation_settings["openai settings"]["openai_logit_bias"]
+        Translator.openai_max_tokens = JsonHandler.current_translation_settings["openai settings"]["openai_max_tokens"]
+        Translator.openai_presence_penalty = float(JsonHandler.current_translation_settings["openai settings"]["openai_presence_penalty"])
+        Translator.openai_frequency_penalty = float(JsonHandler.current_translation_settings["openai settings"]["openai_frequency_penalty"])
+
+        Translator.gemini_model = JsonHandler.current_translation_settings["gemini settings"]["gemini_model"]
+        Translator.gemini_prompt = JsonHandler.current_translation_settings["gemini settings"]["gemini_prompt"]
+        Translator.gemini_temperature = float(JsonHandler.current_translation_settings["gemini settings"]["gemini_temperature"])
+        Translator.gemini_top_p = JsonHandler.current_translation_settings["gemini settings"]["gemini_top_p"]
+        Translator.gemini_top_k = JsonHandler.current_translation_settings["gemini settings"]["gemini_top_k"]
+        Translator.gemini_candidate_count = JsonHandler.current_translation_settings["gemini settings"]["gemini_candidate_count"]
+        Translator.gemini_stream = bool(JsonHandler.current_translation_settings["gemini settings"]["gemini_stream"])
+        Translator.gemini_stop_sequences = JsonHandler.current_translation_settings["gemini settings"]["gemini_stop_sequences"]
+        Translator.gemini_max_output_tokens = JsonHandler.current_translation_settings["gemini settings"]["gemini_max_output_tokens"]
+
+        Translator.deepl_context = JsonHandler.current_translation_settings["deepl settings"]["deepl_context"]
+        Translator.deepl_split_sentences = JsonHandler.current_translation_settings["deepl settings"]["deepl_split_sentences"]
+        Translator.deepl_preserve_formatting = JsonHandler.current_translation_settings["deepl settings"]["deepl_preserve_formatting"]
+        Translator.deepl_formality = JsonHandler.current_translation_settings["deepl settings"]["deepl_formality"]
+
+        exception_dict = {
+            "openai": (OpenAIAuthenticationError, OpenAIInternalServerError, OpenAIRateLimitError, OpenAIAPITimeoutError, OpenAIAPIConnectionError, OpenAIAPIStatusError),
+            "gemini": GoogleAPIError,
+            "deepl": DeepLException
+        }
         
-        Logger.log_barrier()
-        Logger.log_action("Kijiku Activated, LLM Type : " + Kijiku.LLM_TYPE)
-        Logger.log_barrier()
-        Logger.log_action("Settings are as follows : ")
-        Logger.log_barrier()
-
-        JsonHandler.print_kijiku_rules()
-
-        Kijiku.prompt_assembly_mode = int(JsonHandler.current_kijiku_rules["base kijiku settings"]["prompt_assembly_mode"])
-        Kijiku.number_of_lines_per_batch = int(JsonHandler.current_kijiku_rules["base kijiku settings"]["number_of_lines_per_batch"])
-        Kijiku.sentence_fragmenter_mode = int(JsonHandler.current_kijiku_rules["base kijiku settings"]["sentence_fragmenter_mode"])
-        Kijiku.je_check_mode = int(JsonHandler.current_kijiku_rules["base kijiku settings"]["je_check_mode"])
-        Kijiku.num_of_malform_retries = int(JsonHandler.current_kijiku_rules["base kijiku settings"]["number_of_malformed_batch_retries"])
-        Kijiku.max_batch_duration = float(JsonHandler.current_kijiku_rules["base kijiku settings"]["batch_retry_timeout"])
-        Kijiku.num_concurrent_batches = int(JsonHandler.current_kijiku_rules["base kijiku settings"]["number_of_concurrent_batches"])
-
-        Kijiku._semaphore = asyncio.Semaphore(Kijiku.num_concurrent_batches)
-
-        Kijiku.openai_model = JsonHandler.current_kijiku_rules["openai settings"]["openai_model"]
-        Kijiku.openai_system_message = JsonHandler.current_kijiku_rules["openai settings"]["openai_system_message"]
-        Kijiku.openai_temperature = float(JsonHandler.current_kijiku_rules["openai settings"]["openai_temperature"])
-        Kijiku.openai_top_p = float(JsonHandler.current_kijiku_rules["openai settings"]["openai_top_p"])
-        Kijiku.openai_n = int(JsonHandler.current_kijiku_rules["openai settings"]["openai_n"])
-        Kijiku.openai_stream = bool(JsonHandler.current_kijiku_rules["openai settings"]["openai_stream"])
-        Kijiku.openai_stop = JsonHandler.current_kijiku_rules["openai settings"]["openai_stop"]
-        Kijiku.openai_logit_bias = JsonHandler.current_kijiku_rules["openai settings"]["openai_logit_bias"]
-        Kijiku.openai_max_tokens = JsonHandler.current_kijiku_rules["openai settings"]["openai_max_tokens"]
-        Kijiku.openai_presence_penalty = float(JsonHandler.current_kijiku_rules["openai settings"]["openai_presence_penalty"])
-        Kijiku.openai_frequency_penalty = float(JsonHandler.current_kijiku_rules["openai settings"]["openai_frequency_penalty"])
-
-        Kijiku.gemini_model = JsonHandler.current_kijiku_rules["gemini settings"]["gemini_model"]
-        Kijiku.gemini_prompt = JsonHandler.current_kijiku_rules["gemini settings"]["gemini_prompt"]
-        Kijiku.gemini_temperature = float(JsonHandler.current_kijiku_rules["gemini settings"]["gemini_temperature"])
-        Kijiku.gemini_top_p = JsonHandler.current_kijiku_rules["gemini settings"]["gemini_top_p"]
-        Kijiku.gemini_top_k = JsonHandler.current_kijiku_rules["gemini settings"]["gemini_top_k"]
-        Kijiku.gemini_candidate_count = JsonHandler.current_kijiku_rules["gemini settings"]["gemini_candidate_count"]
-        Kijiku.gemini_stream = bool(JsonHandler.current_kijiku_rules["gemini settings"]["gemini_stream"])
-        Kijiku.gemini_stop_sequences = JsonHandler.current_kijiku_rules["gemini settings"]["gemini_stop_sequences"]
-        Kijiku.gemini_max_output_tokens = JsonHandler.current_kijiku_rules["gemini settings"]["gemini_max_output_tokens"]
-
-
-        if(Kijiku.LLM_TYPE == "openai"):
-            Kijiku.decorator_to_use = backoff.on_exception(backoff.expo, max_time=lambda: Kijiku.get_max_batch_duration(), exception=(AuthenticationError, InternalServerError, RateLimitError, APITimeoutError), on_backoff=lambda details: Kijiku.log_retry(details), on_giveup=lambda details: Kijiku.log_failure(details), raise_on_giveup=False)
-
-        else:
-            Kijiku.decorator_to_use = backoff.on_exception(backoff.expo, max_time=lambda: Kijiku.get_max_batch_duration(), exception=(Exception), on_backoff=lambda details: Kijiku.log_retry(details), on_giveup=lambda details: Kijiku.log_failure(details), raise_on_giveup=False)
+        Translator.decorator_to_use = backoff.on_exception(
+            backoff.expo,
+            max_time=lambda: Translator.get_max_batch_duration(),
+            exception=exception_dict.get(Translator.TRANSLATION_METHOD, None),
+            on_backoff=lambda details: Translator.log_retry(details),
+            on_giveup=lambda details: Translator.log_failure(details),
+            raise_on_giveup=False
+        )
 
         Toolkit.clear_console()
 
-        Logger.log_barrier()
-        Logger.log_action("Starting Prompt Building")
-        Logger.log_barrier()
-
-        Kijiku.build_translation_batches()
+        logging.info("Starting Prompt Building...")
 
-        model = JsonHandler.current_kijiku_rules["openai settings"]["openai_model"] if Kijiku.LLM_TYPE == "openai" else JsonHandler.current_kijiku_rules["gemini settings"]["gemini_model"]
+        Translator.build_translation_batches()
+        
+        translation_methods = {
+            "openai": JsonHandler.current_translation_settings["openai settings"]["openai_model"],
+            "gemini": JsonHandler.current_translation_settings["gemini settings"]["gemini_model"],
+            "deepl": "deepl"
+        }
+        
+        model = translation_methods[Translator.TRANSLATION_METHOD]
 
-        await Kijiku.handle_cost_estimate_prompt(model, omit_prompt=is_webgui)
+        await Translator.handle_cost_estimate_prompt(model, omit_prompt=is_webgui or Translator.is_cli)
 
         Toolkit.clear_console()
 
-        Logger.log_barrier()
-        
-        Logger.log_action("Starting Translation...", output=not is_webgui)
-        Logger.log_barrier()
+        logging.info("Starting Translation...")
 
         ## requests to run asynchronously
-        async_requests = Kijiku.build_async_requests(model)
+        async_requests = Translator.build_async_requests(model)
 
         ## Use asyncio.gather to run tasks concurrently/asynchronously and wait for all of them to complete
         results = await asyncio.gather(*async_requests)
 
-        Logger.log_barrier()
-        Logger.log_action("Translation Complete!", output=not is_webgui)
-
-        Logger.log_barrier()
-        Logger.log_action("Starting Redistribution...", output=not is_webgui)
-
-        Logger.log_barrier()
+        logging.info("Redistributing Translated Text...")
 
         ## Sort results based on the index to maintain order
         sorted_results = sorted(results, key=lambda x: x[0])
 
         ## Redistribute the sorted results
-        for index, translated_prompt, translated_message in sorted_results:
-            Kijiku.redistribute(translated_prompt, translated_message)
+        for _, translated_prompt, translated_message in sorted_results:
+            Translator.redistribute(translated_prompt, translated_message)
 
         ## try to pair the text for j-e checking if the mode is 2
-        if(Kijiku.je_check_mode == 2):
-            Kijiku.je_check_text = Kijiku.fix_je()
+        if(Translator.je_check_mode == 2):
+            Translator.je_check_text = Translator.fix_je()
 
         Toolkit.clear_console()
 
-        Logger.log_action("Done!", output=not is_webgui)
-        Logger.log_barrier()
-
-        ## assemble error text based of the error list
-        Kijiku.error_text = Logger.errors
+        logging.info("Done!")
 
 ##-------------------start-of-build_async_requests()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
     
@@ -502,19 +512,34 @@ class Kijiku:
         """
 
         async_requests = []
+        
+        translation_batches_methods = {
+            "openai": Translator.openai_translation_batches,
+            "gemini": Translator.gemini_translation_batches,
+            "deepl": Translator.deepl_translation_batches
+        }
+        
+        translation_batches = translation_batches_methods[Translator.TRANSLATION_METHOD]
+        batch_length = len(translation_batches)
+        
+        if(Translator.TRANSLATION_METHOD != "deepl"):
 
-        translation_batches = Kijiku.openai_translation_batches if Kijiku.LLM_TYPE == "openai" else Kijiku.gemini_translation_batches
-
-        for i in range(0, len(translation_batches), 2):
-
-            instructions = translation_batches[i]
-            prompt = translation_batches[i+1]
-
-            assert isinstance(instructions, SystemTranslationMessage) or isinstance(instructions, str)
-            assert isinstance(prompt, ModelTranslationMessage) or isinstance(prompt, str)
+            for i in range(0, batch_length, 2):
+                instructions = translation_batches[i]
+                prompt = translation_batches[i+1]
+            
+                assert isinstance(instructions, (SystemTranslationMessage, str))
+                assert isinstance(prompt, (ModelTranslationMessage, str))
+            
+                async_requests.append(Translator.handle_translation(model, i, batch_length, prompt, instructions))
 
-            async_requests.append(Kijiku.handle_translation(model, i, len(translation_batches), instructions, prompt))
+        else:
+            for i, batch in enumerate(translation_batches):
 
+                assert isinstance(batch, str)
+            
+                async_requests.append(Translator.handle_translation(model, i, batch_length, batch, None))
+        
         return async_requests
 
 ##-------------------start-of-generate_text_to_translate_batches()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
@@ -537,41 +562,36 @@ class Kijiku:
 
         prompt = []
         non_word_pattern = re.compile(r'^[\W_\s\n-]+$')
+        special_chars = ["▼", "△", "◇"]
+        quotes = ["「", "」", "『", "』", "【", "】", "\"", "'"]
+        part_chars = ["１","２","３","４","５","６","７","８","９", " "]
+        
+        while(index < len(Translator.text_to_translate)):
 
-        while(index < len(Kijiku.text_to_translate)):
-
-            sentence = Kijiku.text_to_translate[index]
-            stripped_sentence = sentence.strip()
+            sentence = Translator.text_to_translate[index].strip()
             lowercase_sentence = sentence.lower()
-
-            has_quotes = any(char in sentence for char in ["「", "」", "『", "』", "【", "】", "\"", "'"])
+        
+            has_quotes = any(char in sentence for char in quotes)
             is_part_in_sentence = "part" in lowercase_sentence
-
-            if(len(prompt) < Kijiku.number_of_lines_per_batch):
-
-                if(any(char in sentence for char in ["▼", "△", "◇"])):
+            is_special_char = any(char in sentence for char in special_chars)
+            is_part_char = all(char in sentence for char in part_chars)
+        
+            if(len(prompt) < Translator.number_of_lines_per_batch):
+                if(is_special_char or is_part_in_sentence or is_part_char):
                     prompt.append(f'{sentence}\n')
-                    Logger.log_action(f"Sentence : {sentence}, Sentence is a pov change... adding to prompt.")
-
-                elif(stripped_sentence == ''):
-                    Logger.log_action(f"Sentence : {sentence} is empty... skipping.")
+                    logging.debug(f"Sentence : {sentence}, Sentence is a pov change or part marker... adding to prompt.")
 
-                elif(is_part_in_sentence or all(char in ["１","２","３","４","５","６","７","８","９", " "] for char in sentence)):
-                    prompt.append(f'{sentence}\n') 
-                    Logger.log_action(f"Sentence : {sentence}, Sentence is part marker... adding to prompt.")
+                elif(non_word_pattern.match(sentence) or KatakanaUtil.is_punctuation(sentence) and not has_quotes):
+                    logging.debug(f"Sentence : {sentence}, Sentence is punctuation... skipping.")
 
-                elif(non_word_pattern.match(sentence) or KatakanaUtil.is_punctuation(stripped_sentence) and not has_quotes):
-                    Logger.log_action(f"Sentence : {sentence}, Sentence is punctuation... skipping.")
-                    
-                else:
+                elif(sentence):
                     prompt.append(f'{sentence}\n')
-                    Logger.log_action(f"Sentence : {sentence}, Sentence is a valid sentence... adding to prompt.")
-
+                    logging.debug(f"Sentence : {sentence}, Sentence is a valid sentence... adding to prompt.")
             else:
                 return prompt, index
-
+        
             index += 1
-
+        
         return prompt, index
     
 ##-------------------start-of-build_translation_batches()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
@@ -587,42 +607,53 @@ class Kijiku:
 
         i = 0
 
-        while i < len(Kijiku.text_to_translate):
+        while i < len(Translator.text_to_translate):
 
-            batch, i = Kijiku.generate_text_to_translate_batches(i)
+            batch, i = Translator.generate_text_to_translate_batches(i)
             batch = ''.join(batch)
 
-            if(Kijiku.LLM_TYPE == 'openai'):
+            if(Translator.TRANSLATION_METHOD == 'openai'):
 
-                if(Kijiku.prompt_assembly_mode == 1):
-                    system_msg = SystemTranslationMessage(content=str(Kijiku.openai_system_message))
+                if(Translator.prompt_assembly_mode == 1):
+                    system_msg = SystemTranslationMessage(content=str(Translator.openai_system_message))
                 else:
-                    system_msg = SystemTranslationMessage(content=str(Kijiku.openai_system_message))
+                    system_msg = SystemTranslationMessage(content=str(Translator.openai_system_message))
 
-                Kijiku.openai_translation_batches.append(system_msg)
+                Translator.openai_translation_batches.append(system_msg)
                 model_msg = ModelTranslationMessage(content=batch)
-                Kijiku.openai_translation_batches.append(model_msg)
+                Translator.openai_translation_batches.append(model_msg)
+
+            elif(Translator.TRANSLATION_METHOD == 'gemini'):
+                Translator.gemini_translation_batches.append(Translator.gemini_prompt)
+                Translator.gemini_translation_batches.append(batch)
 
             else:
-                Kijiku.gemini_translation_batches.append(Kijiku.gemini_prompt)
-                Kijiku.gemini_translation_batches.append(batch)
+                Translator.deepl_translation_batches.append(batch)
 
-        Logger.log_barrier()
-        Logger.log_action("Built Messages : ")
-        Logger.log_barrier()
+        logging_message = "Built Messages: \n\n"
+
+        batches_to_iterate = {
+            "openai": Translator.openai_translation_batches,
+            "gemini": Translator.gemini_translation_batches,
+            "deepl": Translator.deepl_translation_batches
+        }
 
         i = 0
 
-        for message in (Kijiku.openai_translation_batches if Kijiku.LLM_TYPE == 'openai' else Kijiku.gemini_translation_batches):
+        batches = batches_to_iterate[Translator.TRANSLATION_METHOD]
+
+        for message in batches:
 
             i+=1
 
-            message = str(message) if Kijiku.LLM_TYPE == 'gemini' else message.content # type: ignore
+            message = str(message) if Translator.TRANSLATION_METHOD != 'openai' else message.content # type: ignore
+
+            if(i % 2 == 1 and Translator.TRANSLATION_METHOD != 'deepl'):
+                logging_message += "\n" "------------------------" "\n"
 
-            if(i % 2 == 1):
-                Logger.log_barrier()
+            logging_message += message + "\n"
 
-            Logger.log_action(message)
+        logging.debug(logging_message)
 
 ##-------------------start-of-handle_cost_estimate_prompt()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
@@ -642,39 +673,43 @@ class Kijiku:
         
         """ 
 
-        translation_instructions = Kijiku.openai_system_message if Kijiku.LLM_TYPE == "openai" else Kijiku.gemini_prompt
+        translation_instructions_methods = {
+            "openai": Translator.openai_system_message,
+            "gemini": Translator.gemini_prompt,
+            "deepl": None,
+        }
+        
+        translation_instructions = translation_instructions_methods[Translator.TRANSLATION_METHOD]
 
         ## get cost estimate and confirm
-        num_tokens, min_cost, model = EasyTL.calculate_cost(text=Kijiku.text_to_translate, service=Kijiku.LLM_TYPE, model=model,translation_instructions=translation_instructions)
+        num_tokens, min_cost, model = EasyTL.calculate_cost(text=Translator.text_to_translate, service=Translator.TRANSLATION_METHOD, model=model,translation_instructions=translation_instructions)
 
         print("Note that the cost estimate is not always accurate, and may be higher than the actual cost. However cost calculation now includes output tokens.\n")
 
-        Logger.log_barrier()
-        Logger.log_action("Calculating cost")
-        Logger.log_barrier()
-
-        if(Kijiku.LLM_TYPE == "gemini"):
-            Logger.log_action(f"As of Kudasai {Toolkit.CURRENT_VERSION}, Gemini Pro 1.0 is free to use under 15 requests per minute, Gemini Pro 1.5 is free to use under 2 requests per minute. Requests correspond to number_of_current_batches in kijiku_settings.", output=True, omit_timestamp=True)
+        if(Translator.TRANSLATION_METHOD == "gemini"):
+            logging.info(f"As of Kudasai {Toolkit.CURRENT_VERSION}, Gemini Pro 1.0 is free to use under 15 requests per minute, Gemini Pro 1.5 is free to use under 2 requests per minute. Requests correspond to number_of_current_batches in the translation settings.")
         
-        Logger.log_action("Estimated number of tokens : " + str(num_tokens), output=True, omit_timestamp=True)
-        Logger.log_action("Estimated minimum cost : " + str(min_cost) + " USD", output=True, omit_timestamp=True)
-        Logger.log_barrier()
+        logging.info("Estimated number of tokens : " + str(num_tokens))
+        logging.info("Estimated minimum cost : " + str(min_cost) + " USD")
 
         if(not omit_prompt):
             if(input("\nContinue? (1 for yes or 2 for no) : ") == "1"):
-                Logger.log_action("User confirmed translation.")
+                logging.info("User confirmed translation.")
 
             else:
-                Logger.log_action("User cancelled translation.")
-                Logger.push_batch()
-                exit()
+                logging.info("User cancelled translation.")
+                FileEnsurer.exit_kudasai()
 
         return model
     
 ##-------------------start-of-handle_translation()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
     
     @staticmethod
-    async def handle_translation(model:str, index:int, length:int, translation_instructions:typing.Union[str, SystemTranslationMessage], translation_prompt:typing.Union[str, ModelTranslationMessage]) -> tuple[int, str, str]:
+    async def handle_translation(model:str, 
+                                 batch_index:int,
+                                 length_of_batch:int, 
+                                 text_to_translate:typing.Union[str, ModelTranslationMessage],
+                                 translation_instructions:typing.Union[str, SystemTranslationMessage, None]) -> tuple[int, str, str]: 
 
         """
 
@@ -682,20 +717,20 @@ class Kijiku:
 
         Parameters:
         model (string) : The model of the service used to translate the text.
-        index (int) : The index of the translation batch.
-        length (int) : The length of the translation batch.
-        translation_instructions (typing.Union[str, Message]) : The translation instructions.
-        translation_prompt (typing.Union[str, Message]) : The translation prompt.
+        batch_index (int) : Which batch we are currently on.
+        length_of_batch (int) : How long the batches are.
+        text_to_translate (typing.Union[str, ModelTranslationMessage]) : The text to translate.
+        translation_instructions (typing.Union[str, SystemTranslationMessage, None]) : The translation instructions.
 
         Returns:
-        index (int) : The index of the translation batch.
-        translation_prompt (typing.Union[str, Message]) : The translation prompt.
-        translated_message (str) : The translated message.
+        batch_index (int) : The batch index.
+        text_to_translate (str) : The text to translate.
+        translated_text (str) : The translated text
 
         """
 
         ## Basically limits the number of concurrent batches
-        async with Kijiku._semaphore:
+        async with Translator._semaphore:
             num_tries = 0
 
             while True:
@@ -704,72 +739,93 @@ class Kijiku:
                 if(FileEnsurer.do_interrupt == True):
                     raise Exception("Interrupted by user.")
 
-                message_number = (index // 2) + 1
-                Logger.log_action(f"Trying translation for batch {message_number} of {length//2}...", output=True)
+                batch_number = (batch_index // 2) + 1
+
+                logging.info(f"Trying translation for batch {batch_number} of {length_of_batch//2}...")
 
                 try:
 
-                    if(Kijiku.LLM_TYPE == "openai"):
-                        translated_message = await EasyTL.openai_translate_async(text=translation_prompt,
-                                                                                decorator=Kijiku.decorator_to_use,
-                                                                                translation_instructions=translation_instructions,
-                                                                                model=model,
-                                                                                temperature=Kijiku.openai_temperature,
-                                                                                top_p=Kijiku.openai_top_p,
-                                                                                stop=Kijiku.openai_stop,
-                                                                                max_tokens=Kijiku.openai_max_tokens,
-                                                                                presence_penalty=Kijiku.openai_presence_penalty,
-                                                                                frequency_penalty=Kijiku.openai_frequency_penalty)
-
-                    else:
-
-                        assert isinstance(translation_prompt, str)
-
-                        translated_message = await EasyTL.gemini_translate_async(text=translation_prompt,
-                                                                                 decorator=Kijiku.decorator_to_use,
-                                                                                 model=model,
-                                                                                 temperature=Kijiku.gemini_temperature,
-                                                                                 top_p=Kijiku.gemini_top_p,
-                                                                                 top_k=Kijiku.gemini_top_k,
-                                                                                 stop_sequences=Kijiku.gemini_stop_sequences,
-                                                                                 max_output_tokens=Kijiku.gemini_max_output_tokens)
+                    translation_methods = {
+                        "openai": EasyTL.openai_translate_async,
+                        "gemini": EasyTL.gemini_translate_async,
+                        "deepl": EasyTL.deepl_translate_async
+                    }
+                    
+                    translation_params = {
+                        "openai": {
+                            "text": text_to_translate,
+                            "decorator": Translator.decorator_to_use,
+                            "translation_instructions": translation_instructions,
+                            "model": model,
+                            "temperature": Translator.openai_temperature,
+                            "top_p": Translator.openai_top_p,
+                            "stop": Translator.openai_stop,
+                            "max_tokens": Translator.openai_max_tokens,
+                            "presence_penalty": Translator.openai_presence_penalty,
+                            "frequency_penalty": Translator.openai_frequency_penalty
+                        },
+                        "gemini": {
+                            "text": text_to_translate,
+                            "decorator": Translator.decorator_to_use,
+                            "model": model,
+                            "temperature": Translator.gemini_temperature,
+                            "top_p": Translator.gemini_top_p,
+                            "top_k": Translator.gemini_top_k,
+                            "stop_sequences": Translator.gemini_stop_sequences,
+                            "max_output_tokens": Translator.gemini_max_output_tokens
+                        },
+                        "deepl": {
+                            "text": text_to_translate,
+                            "decorator": Translator.decorator_to_use,
+                            "context": Translator.deepl_context,
+                            "split_sentences": Translator.deepl_split_sentences,
+                            "preserve_formatting": Translator.deepl_preserve_formatting,
+                            "formality": Translator.deepl_formality
+                        }
+                    }
+                    
+                    assert isinstance(text_to_translate, ModelTranslationMessage if Translator.TRANSLATION_METHOD == "openai" else str)
+                    
+                    translated_message = await translation_methods[Translator.TRANSLATION_METHOD](**translation_params[Translator.TRANSLATION_METHOD])
 
                 ## will only occur if the max_batch_duration is exceeded, so we just return the untranslated text
                 except MaxBatchDurationExceededException:
 
-                    Logger.log_error(f"Batch {message_number} of {length//2} was not translated due to exceeding the max request duration, returning the untranslated text...", output=True)
+                    logging.error(f"Batch {batch_number} of {length_of_batch//2} was not translated due to exceeding the max request duration, returning the untranslated text...")
                     break
 
                 ## do not even bother if not a gpt 4 model, because gpt-3 seems unable to format properly
                 ## since gemini is free, we can just try again if it's malformed
-                if("gpt-4" not in model and Kijiku.LLM_TYPE != "gemini"): 
+                ## deepl should produce properly formatted text so we don't need to check
+                if("gpt-4" not in model and Translator.TRANSLATION_METHOD == "openai"):
                     break
 
-                if(await Kijiku.check_if_translation_is_good(translated_message, translation_prompt)):
-                    Logger.log_action(f"Translation for batch {message_number} of {length//2} successful!", output=True)
+                if(await Translator.check_if_translation_is_good(translated_message, text_to_translate)): # type: ignore
                     break
 
-                if(num_tries >= Kijiku.num_of_malform_retries):
-                    Logger.log_action(f"Batch {message_number} of {length//2} was malformed, but exceeded the maximum number of retries, Translation successful!", output=True)
+                if(num_tries >= Translator.num_of_malform_retries):
+                    logging.warning(f"Batch {batch_number} of {length_of_batch//2} was malformed but exceeded the max number of retries ({Translator.num_of_malform_retries})")
                     break
 
                 else:
                     num_tries += 1
-                    Logger.log_error(f"Batch {message_number} of {length//2} was malformed, retrying...", output=True)
-                    Kijiku.num_occurred_malformed_batches += 1
+                    logging.warning(f"Batch {batch_number} of {length_of_batch//2} was malformed, retrying...")
+                    Translator.num_occurred_malformed_batches += 1
 
-            if(isinstance(translation_prompt, ModelTranslationMessage)):
-                translation_prompt = translation_prompt.content
+            if(isinstance(text_to_translate, ModelTranslationMessage)):
+                text_to_translate = text_to_translate.content
 
             if(isinstance(translated_message, typing.List)):
-                translated_message = ''.join(translated_message)
+                translated_message = ''.join(translated_message) # type: ignore
+
+            logging.info(f"Translation for batch {batch_number} of {length_of_batch//2} completed.")
 
-            return index, translation_prompt, translated_message
+            return batch_index, text_to_translate, translated_message # type: ignore
     
 ##-------------------start-of-check_if_translation_is_good()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
     @staticmethod
-    async def check_if_translation_is_good(translated_message:typing.Union[typing.List[str], str], translation_prompt:typing.Union[ModelTranslationMessage, str]) -> bool:
+    async def check_if_translation_is_good(translated_message:typing.Union[typing.List[str], str], text_to_translate:typing.Union[ModelTranslationMessage, str]) -> bool:
 
         """
         
@@ -777,66 +833,61 @@ class Kijiku:
 
         Parameters:
         translated_message (str) : the translated message.
-        translation_prompt (typing.Union[str, Message]) : the translation prompt.
+        text_to_translate (typing.Union[str, Message]) : the translation prompt.
 
         Returns:
         is_valid (bool) : whether or not the translation is valid.
 
         """
 
-        if(not isinstance(translation_prompt, str)):
-            prompt = translation_prompt.content
+        if(not isinstance(text_to_translate, str)):
+            prompt = text_to_translate.content
 
         else:
-            prompt = translation_prompt
+            prompt = text_to_translate
 
         if(isinstance(translated_message, list)):
             translated_message = ''.join(translated_message)
             
-        is_valid = False
-
         jap = [line for line in prompt.split('\n') if line.strip()]  ## Remove blank lines
         eng = [line for line in translated_message.split('\n') if line.strip()]  ## Remove blank lines
-
-        if(len(jap) == len(eng)):
-            is_valid = True
     
-        return is_valid
+        return len(jap) == len(eng)
     
 ##-------------------start-of-redistribute()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
     @staticmethod
-    def redistribute(translation_prompt:typing.Union[Message, str], translated_message:str) -> None:
+    def redistribute(text_to_translate:typing.Union[Message, str], translated_message:str) -> None:
 
         """
 
         Puts translated text back into the text file.
 
         Parameters:
-        translation_prompt (typing.Union[str, Message]) : the translation prompt.
+        text_to_translate (typing.Union[str, Message]) : the translation prompt.
         translated_message (str) : the translated message.
 
         """
 
-        if(not isinstance(translation_prompt, str)):
-            prompt = translation_prompt.content
+        if(not isinstance(text_to_translate, str)):
+            prompt = text_to_translate.content
 
         else:
-            prompt = translation_prompt
+            prompt = text_to_translate
 
         ## Separates with hyphens if the mode is 1 
-        if(Kijiku.je_check_mode == 1):
+        if(Translator.je_check_mode == 1):
 
-            Kijiku.je_check_text.append("\n-------------------------\n"+ prompt + "\n\n")
-            Kijiku.je_check_text.append(translated_message + '\n')
+            Translator.je_check_text.append("\n-------------------------\n"+ prompt + "\n\n")
+            Translator.je_check_text.append(translated_message + '\n')
         
         ## Mode two tries to pair the text for j-e checking, see fix_je() for more details
-        elif(Kijiku.je_check_mode == 2):
-            Kijiku.je_check_text.append(prompt)
-            Kijiku.je_check_text.append(translated_message)
+        elif(Translator.je_check_mode == 2):
+            Translator.je_check_text.append(prompt)
+            Translator.je_check_text.append(translated_message)
 
         ## mode 1 is the default mode, uses regex and other nonsense to split sentences
-        if(Kijiku.sentence_fragmenter_mode == 1): 
+        if(Translator.sentence_fragmenter_mode == 1): 
 
             sentences = re.findall(r"(.*?(?:(?:\"|\'|-|~|!|\?|%|\(|\)|\.\.\.|\.|---|\[|\])))(?:\s|$)", translated_message)
 
@@ -859,17 +910,17 @@ class Kijiku:
                     build_string += f" {sentence}"
                     continue
 
-                Kijiku.translated_text.append(sentence + '\n')
+                Translator.translated_text.append(sentence + '\n')
 
-            for i in range(len(Kijiku.translated_text)):
-                if Kijiku.translated_text[i] in patched_sentences:
-                    index = patched_sentences.index(Kijiku.translated_text[i])
-                    Kijiku.translated_text[i] = patched_sentences[index]
+            for i in range(len(Translator.translated_text)):
+                if Translator.translated_text[i] in patched_sentences:
+                    index = patched_sentences.index(Translator.translated_text[i])
+                    Translator.translated_text[i] = patched_sentences[index]
 
         ## mode 2 just assumes the LLM formatted it properly
-        elif(Kijiku.sentence_fragmenter_mode == 2):
+        elif(Translator.sentence_fragmenter_mode == 2):
             
-            Kijiku.translated_text.append(translated_message + '\n\n')
+            Translator.translated_text.append(translated_message + '\n\n')
         
 ##-------------------start-of-fix_je()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
@@ -890,36 +941,30 @@ class Kijiku:
         i = 1
         final_list = []
 
-        while i < len(Kijiku.je_check_text):
-            jap = Kijiku.je_check_text[i-1].split('\n')
-            eng = Kijiku.je_check_text[i].split('\n')
+        while(i < len(Translator.je_check_text)):
+            jap = Translator.je_check_text[i-1].split('\n')
+            eng = Translator.je_check_text[i].split('\n')
 
-            jap = [line for line in jap if line.strip()]  ## Remove blank lines
-            eng = [line for line in eng if line.strip()]  ## Remove blank lines    
+            jap = [line for line in jap if(line.strip())]  # Remove blank lines
+            eng = [line for line in eng if(line.strip())]  # Remove blank lines    
 
             final_list.append("-------------------------\n")
 
             if(len(jap) == len(eng)):
-
-                for jap_line,eng_line in zip(jap,eng):
-                    if(jap_line and eng_line): ## check if jap_line and eng_line aren't blank
+                for(jap_line, eng_line) in zip(jap, eng):
+                    if(jap_line and eng_line):  # check if jap_line and eng_line aren't blank
                         final_list.append(jap_line + '\n\n')
                         final_list.append(eng_line + '\n\n')
-
                         final_list.append("--------------------------------------------------\n")
-     
-
             else:
-
-                final_list.append(Kijiku.je_check_text[i-1] + '\n\n')
-                final_list.append(Kijiku.je_check_text[i] + '\n\n')
-
+                final_list.append(Translator.je_check_text[i-1] + '\n\n')
+                final_list.append(Translator.je_check_text[i] + '\n\n')
                 final_list.append("--------------------------------------------------\n")
 
-            i+=2
+            i += 2
 
         return final_list
-
+    
 ##-------------------start-of-assemble_results()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
     
     @staticmethod
@@ -927,7 +972,7 @@ class Kijiku:
 
         """
 
-        Generates the Kijiku translation print result, does not directly output/return, but rather sets Kijiku.translation_print_result to the output.
+        Generates the Translator translation print result, does not directly output/return, but rather sets Translator.translation_print_result to the output.
 
         Parameters:
         time_start (float) : When the translation started.
@@ -937,24 +982,24 @@ class Kijiku:
 
         result = (
             f"Time Elapsed : {Toolkit.get_elapsed_time(time_start, time_end)}\n"
-            f"Number of malformed batches : {Kijiku.num_occurred_malformed_batches}\n\n"
+            f"Number of malformed batches : {Translator.num_occurred_malformed_batches}\n\n"
             f"Debug text have been written to : {FileEnsurer.debug_log_path}\n"
             f"J->E text have been written to : {FileEnsurer.je_check_path}\n"
             f"Translated text has been written to : {FileEnsurer.translated_text_path}\n"
             f"Errors have been written to : {FileEnsurer.error_log_path}\n"
         )
         
-        Kijiku.translation_print_result = result
+        Translator.translation_print_result = result
 
-##-------------------start-of-write_kijiku_results()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+##-------------------start-of-write_translator_results()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
     @staticmethod
     @permission_error_decorator()
-    def write_kijiku_results() -> None:
+    def write_translator_results() -> None:
 
         """
         
-        This function is called to write the results of the Kijiku translation module to the output directory.
+        This function is called to write the results of the Translator module to the output directory.
 
         """
 
@@ -962,27 +1007,23 @@ class Kijiku:
         FileEnsurer.standard_create_directory(FileEnsurer.output_dir)
 
         with open(FileEnsurer.error_log_path, 'a+', encoding='utf-8') as file:
-            file.writelines(Kijiku.error_text)
+            file.writelines(Translator.error_text)
 
         with open(FileEnsurer.je_check_path, 'w', encoding='utf-8') as file:
-            file.writelines(Kijiku.je_check_text)
+            file.writelines(Translator.je_check_text)
 
         with open(FileEnsurer.translated_text_path, 'w', encoding='utf-8') as file:
-            file.writelines(Kijiku.translated_text)
+            file.writelines(Translator.translated_text)
 
         ## Instructions to create a copy of the output for archival
         FileEnsurer.standard_create_directory(FileEnsurer.archive_dir)
 
         timestamp = Toolkit.get_timestamp(is_archival=True)
 
-        ## pushes the tl debug log to the file without clearing the file
-        Logger.push_batch()
-        Logger.clear_batch()
-
-        list_of_result_tuples = [('kijiku_translated_text', Kijiku.translated_text), 
-                                 ('kijiku_je_check_text', Kijiku.je_check_text), 
-                                 ('kijiku_error_log', Kijiku.error_text),
-                                 ('debug_log', FileEnsurer.standard_read_file(Logger.log_file_path))]
+        list_of_result_tuples = [('kudasai_translated_text', Translator.translated_text), 
+                                 ('kudasai_je_check_text', Translator.je_check_text), 
+                                 ('kudasai_error_log', Translator.error_text),
+                                 ('debug_log', FileEnsurer.standard_read_file(FileEnsurer.debug_log_path))]
 
         FileEnsurer.archive_results(list_of_result_tuples, 
-                                    module='kijiku', timestamp=timestamp)
+                                    module='translator', timestamp=timestamp)

modules/gui/gui_json_util.py

@@ -12,16 +12,16 @@ from handlers.json_handler import JsonHandler
 
 class GuiJsonUtil:
 
-    current_kijiku_rules = dict()
+    current_translation_settings = dict()
 
 ##-------------------start-of-fetch_kijiku_setting_key_values()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
     @staticmethod
-    def fetch_kijiku_setting_key_values(header:str, key_name:str) -> str:
+    def fetch_translation_settings_key_values(header:str, key_name:str) -> str:
         
         """
         
-        Fetches the default values for the settings tab from the kijiku_settings.json file.
+        Fetches the default values for the settings tab from the translation_settings.json file.
 
         Parameters:
         key_name (str) : Which value to fetch.
@@ -32,16 +32,16 @@ class GuiJsonUtil:
         """
 
         ## Done this way because if the value is None, it'll be shown as a blank string in the settings tab, which is not what we want.
-        return GuiJsonUtil.current_kijiku_rules[header].get(key_name, "None")
+        return GuiJsonUtil.current_translation_settings[header].get(key_name, "None")
     
-##-------------------start-of-update_kijiku_settings_with_new_values()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+##-------------------start-of-update_translation_settings_with_new_values()---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
     @staticmethod
-    def update_kijiku_settings_with_new_values(gradio_kijiku_rule:gr.File, new_values:typing.List[typing.Tuple[str,str]]) -> None:
+    def update_translation_settings_with_new_values(gradio_translation_settings:gr.File, new_values:typing.List[typing.Tuple[str,str]]) -> None:
 
         """
         
-        Dumps the new values for the settings tab into the kijiku_settings.json file.
+        Dumps the new values for the settings tab into the translation_settings.json file.
 
         Parameters:
         new_values (typing.List[typing.Tuple[str,str]]) : A list of tuples containing the key and value to be updated.
@@ -49,7 +49,7 @@ class GuiJsonUtil:
         """
 
         ## save old json in case of need to revert
-        old_rules = GuiJsonUtil.current_kijiku_rules
+        old_rules = GuiJsonUtil.current_translation_settings
         new_rules = old_rules.copy()
 
         try:
@@ -58,32 +58,32 @@ class GuiJsonUtil:
                 for key, value in new_values:
                     new_rules[header][key] = JsonHandler.convert_to_correct_type(key, str(value))
 
-            JsonHandler.current_kijiku_rules = new_rules
+            JsonHandler.current_translation_settings = new_rules
             JsonHandler.validate_json()
 
             ## validate_json() sets a dict to the invalid placeholder if it's invalid, so if it's that, it's invalid
-            assert JsonHandler.current_kijiku_rules != FileEnsurer.INVALID_KIJIKU_RULES_PLACEHOLDER
+            assert JsonHandler.current_translation_settings != FileEnsurer.INVALID_TRANSLATION_SETTINGS_PLACEHOLDER
 
-            ## so, because of how gradio deals with temp file, we need to both dump into the settings file from FileEnsurer AND the gradio_kijiku_rule file which is stored in the temp folder under AppData
+            ## so, because of how gradio deals with temp file, we need to both dump into the settings file from FileEnsurer AND the gradio_translation_settings file which is stored in the temp folder under AppData
             ## name is the path to the file btw
-            with open(FileEnsurer.config_kijiku_rules_path, "w") as file:
+            with open(FileEnsurer.config_translation_settings_path, "w") as file:
                 json.dump(new_rules, file)
 
-            with open(gradio_kijiku_rule.name, "w") as file: ## type: ignore
+            with open(gradio_translation_settings.name, "w") as file: ## type: ignore
                 json.dump(new_rules, file)
 
-            GuiJsonUtil.current_kijiku_rules = new_rules
+            GuiJsonUtil.current_translation_settings = new_rules
 
         except Exception as e:
 
             ## revert to old data
-            with open(FileEnsurer.config_kijiku_rules_path, "w") as file:
+            with open(FileEnsurer.config_translation_settings_path, "w") as file:
                 json.dump(old_rules, file)
 
-            with open(gradio_kijiku_rule.name, "w") as file: ## type: ignore
+            with open(gradio_translation_settings.name, "w") as file: ## type: ignore
                 json.dump(old_rules, file)
 
-            GuiJsonUtil.current_kijiku_rules = old_rules
+            GuiJsonUtil.current_translation_settings = old_rules
 
             ## throw error so webgui can tell user
             raise e

requirements.txt

@@ -1,5 +1,4 @@
 backoff==2.2.1
-gradio==4.19.2
+gradio==4.20.0
 kairyou==1.5.0
-easytl==0.1.0
-ja_core_news_lg @ https://github.com/explosion/spacy-models/releases/download/ja_core_news_lg-3.7.0/ja_core_news_lg-3.7.0-py3-none-any.whl#sha256=f08eecb4d40523045c9478ce59a67564fd71edd215f32c076fa91dc1f05cc7fd
+easytl==0.4.0-alpha-2

util/openai_model_info/openai_chat_model_info.csv

@@ -1,17 +0,0 @@
-,Batch,Name,Price,Recommended Replacement,Depreciation Date,Shutdown Date (earliest)
-,,,,,,
-,First (depreciated I),gpt-3.5-turbo-0301,$0.0015 / 1K input tokens + $0.0020 / 1K output tokens,gpt-3.5-turbo-0613,"June 13, 2023","June 13, 2024"
-,,gpt-4-0314,$0.03 / 1K input tokens + $0.06 / 1K output tokens,gpt-4-0613,"June 13, 2023","June 13, 2024"
-,,gpt-4-32k-0314,$0.06 / 1K input tokens + $0.12 / 1K output tokens,gpt-4-32k-0613,"June 13, 2023","June 13, 2024"
-,,,,,,
-,Second (depreciated II),gpt-3.5-turbo-0613,$0.0015 / 1K input tokens + $0.0020 / 1K output tokens,gpt-3.5-turbo-1106,"November 6,2023","June 13, 2024"
-,,gpt-3.5-turbo-16k-0613,$0.0030 / 1K input tokens + $0.0040 / 1K output tokens,gpt-3.5-turbo-1106,"November 6,2023","June 13, 2024"
-,,,,,,
-,Third (Outdated),gpt-3.5-turbo-1106,$0.0010 / 1K input tokens + $0.0020 / 1K output tokens,N/A,N/A,N/A
-,,,,,,
-,Fourth (Current),gpt-3.5-turbo-0125,$0.005 / 1K input tokens + $0.0015 / 1K output tokens,N/A,N/A,N/A
-,,gpt-4-0613,$0.03 / 1K input tokens + $0.06 / 1K output tokens,N/A,N/A,N/A
-,,gpt-4-32k-0613,$0.06 / 1K input tokens + $0.012 / 1K output tokens,N/A,N/A,N/A
-,,,,,,
-,Fifth (Future),gpt-4-1106-preview,$0.01 / 1K input tokens + $0.03 / 1K output tokens,N/A,N/A,N/A
-,,gpt-4-0125-preview,$0.01 / 1K input tokens + $0.03 / 1K output tokens,N/A,N/A,N/A

util/openai_model_info/webpage/openai_chat_model_info.html

@@ -1,2 +0,0 @@
-<meta http-equiv="Content-Type" content="text/html; charset=utf-8"><link type="text/css" rel="stylesheet" href="resources/sheet.css" >
-<style type="text/css">.ritz .waffle a { color: inherit; }.ritz .waffle .s1{background-color:#ffffff;text-align:right;color:#000000;font-family:'Arial';font-size:10pt;vertical-align:bottom;white-space:nowrap;direction:ltr;padding:2px 3px 2px 3px;}.ritz .waffle .s0{background-color:#ffffff;text-align:left;color:#000000;font-family:'Arial';font-size:10pt;vertical-align:bottom;white-space:nowrap;direction:ltr;padding:2px 3px 2px 3px;}</style><div class="ritz grid-container" dir="ltr"><table class="waffle" cellspacing="0" cellpadding="0"><thead><tr><th class="row-header freezebar-origin-ltr"></th><th id="0C0" style="width:100px;" class="column-headers-background">A</th><th id="0C1" style="width:141px;" class="column-headers-background">B</th><th id="0C2" style="width:159px;" class="column-headers-background">C</th><th id="0C3" style="width:336px;" class="column-headers-background">D</th><th id="0C4" style="width:182px;" class="column-headers-background">E</th><th id="0C5" style="width:167px;" class="column-headers-background">F</th><th id="0C6" style="width:151px;" class="column-headers-background">G</th></tr></thead><tbody><tr style="height: 20px"><th id="0R0" style="height: 20px;" class="row-headers-background"><div class="row-header-wrapper" style="line-height: 20px">1</div></th><td></td><td class="s0" dir="ltr">Batch</td><td class="s0" dir="ltr">Name</td><td class="s0" dir="ltr">Price</td><td class="s0" dir="ltr">Recommended Replacement</td><td class="s0" dir="ltr">Depreciation Date</td><td class="s0" dir="ltr">Shutdown Date (earliest)</td></tr><tr style="height: 20px"><th id="0R1" style="height: 20px;" class="row-headers-background"><div class="row-header-wrapper" style="line-height: 20px">2</div></th><td></td><td></td><td></td><td></td><td></td><td class="s0" dir="ltr"></td><td class="s0" dir="ltr"></td></tr><tr style="height: 20px"><th id="0R2" style="height: 20px;" class="row-headers-background"><div class="row-header-wrapper" style="line-height: 20px">3</div></th><td></td><td class="s0" dir="ltr">First (depreciated I)</td><td class="s0" dir="ltr">gpt-3.5-turbo-0301</td><td class="s0" dir="ltr">$0.0015 / 1K input tokens + $0.0020 / 1K output tokens</td><td class="s0" dir="ltr">gpt-3.5-turbo-0613</td><td class="s1" dir="ltr">June 13, 2023</td><td class="s1" dir="ltr">June 13, 2024</td></tr><tr style="height: 20px"><th id="0R3" style="height: 20px;" class="row-headers-background"><div class="row-header-wrapper" style="line-height: 20px">4</div></th><td></td><td></td><td class="s0" dir="ltr">gpt-4-0314</td><td class="s0" dir="ltr">$0.03 / 1K input tokens + $0.06 / 1K output tokens</td><td class="s0" dir="ltr">gpt-4-0613</td><td class="s1" dir="ltr">June 13, 2023</td><td class="s1" dir="ltr">June 13, 2024</td></tr><tr style="height: 20px"><th id="0R4" style="height: 20px;" class="row-headers-background"><div class="row-header-wrapper" style="line-height: 20px">5</div></th><td></td><td></td><td class="s0" dir="ltr">gpt-4-32k-0314</td><td class="s0" dir="ltr">$0.06 / 1K input tokens + $0.12 / 1K output tokens</td><td class="s0" dir="ltr">gpt-4-32k-0613</td><td class="s1" dir="ltr">June 13, 2023</td><td class="s1" dir="ltr">June 13, 2024</td></tr><tr style="height: 20px"><th id="0R5" style="height: 20px;" class="row-headers-background"><div class="row-header-wrapper" style="line-height: 20px">6</div></th><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr style="height: 20px"><th id="0R6" style="height: 20px;" class="row-headers-background"><div class="row-header-wrapper" style="line-height: 20px">7</div></th><td></td><td class="s0" dir="ltr">Second (depreciated II)</td><td class="s0" dir="ltr">gpt-3.5-turbo-0613</td><td class="s0" dir="ltr">$0.0015 / 1K input tokens + $0.0020 / 1K output tokens</td><td class="s0" dir="ltr">gpt-3.5-turbo-1106</td><td class="s1" dir="ltr">November 6,2023</td><td class="s1" dir="ltr">June 13, 2024</td></tr><tr style="height: 20px"><th id="0R7" style="height: 20px;" class="row-headers-background"><div class="row-header-wrapper" style="line-height: 20px">8</div></th><td></td><td></td><td class="s0" dir="ltr">gpt-3.5-turbo-16k-0613</td><td class="s0" dir="ltr">$0.0030 / 1K input tokens + $0.0040 / 1K output tokens</td><td class="s0" dir="ltr">gpt-3.5-turbo-1106</td><td class="s1" dir="ltr">November 6,2023</td><td class="s1" dir="ltr">June 13, 2024</td></tr><tr style="height: 20px"><th id="0R8" style="height: 20px;" class="row-headers-background"><div class="row-header-wrapper" style="line-height: 20px">9</div></th><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr style="height: 20px"><th id="0R9" style="height: 20px;" class="row-headers-background"><div class="row-header-wrapper" style="line-height: 20px">10</div></th><td></td><td class="s0" dir="ltr">Third (Outdated)</td><td class="s0" dir="ltr">gpt-3.5-turbo-1106</td><td class="s0" dir="ltr">$0.0010 / 1K input tokens + $0.0020 / 1K output tokens</td><td class="s0" dir="ltr">N/A</td><td class="s0" dir="ltr">N/A</td><td class="s0" dir="ltr">N/A</td></tr><tr style="height: 20px"><th id="0R10" style="height: 20px;" class="row-headers-background"><div class="row-header-wrapper" style="line-height: 20px">11</div></th><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr style="height: 20px"><th id="0R11" style="height: 20px;" class="row-headers-background"><div class="row-header-wrapper" style="line-height: 20px">12</div></th><td></td><td class="s0" dir="ltr">Fourth (Current)</td><td class="s0" dir="ltr">gpt-3.5-turbo-0125</td><td class="s0" dir="ltr">$0.005 / 1K input tokens + $0.0015 / 1K output tokens</td><td class="s0" dir="ltr">N/A</td><td class="s0" dir="ltr">N/A</td><td class="s0" dir="ltr">N/A</td></tr><tr style="height: 20px"><th id="0R12" style="height: 20px;" class="row-headers-background"><div class="row-header-wrapper" style="line-height: 20px">13</div></th><td></td><td></td><td class="s0" dir="ltr">gpt-4-0613</td><td class="s0" dir="ltr">$0.03 / 1K input tokens + $0.06 / 1K output tokens</td><td class="s0" dir="ltr">N/A</td><td class="s0" dir="ltr">N/A</td><td class="s0" dir="ltr">N/A</td></tr><tr style="height: 20px"><th id="0R13" style="height: 20px;" class="row-headers-background"><div class="row-header-wrapper" style="line-height: 20px">14</div></th><td></td><td></td><td class="s0" dir="ltr">gpt-4-32k-0613</td><td class="s0" dir="ltr">$0.06 / 1K input tokens + $0.012 / 1K output tokens</td><td class="s0" dir="ltr">N/A</td><td class="s0" dir="ltr">N/A</td><td class="s0" dir="ltr">N/A</td></tr><tr style="height: 20px"><th id="0R14" style="height: 20px;" class="row-headers-background"><div class="row-header-wrapper" style="line-height: 20px">15</div></th><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr style="height: 20px"><th id="0R15" style="height: 20px;" class="row-headers-background"><div class="row-header-wrapper" style="line-height: 20px">16</div></th><td></td><td class="s0">Fifth (Future)</td><td class="s0">gpt-4-1106-preview</td><td class="s0">$0.01 / 1K input tokens + $0.03 / 1K output tokens</td><td class="s0">N/A</td><td class="s0">N/A</td><td class="s0">N/A</td></tr><tr style="height: 20px"><th id="0R16" style="height: 20px;" class="row-headers-background"><div class="row-header-wrapper" style="line-height: 20px">17</div></th><td></td><td class="s0"></td><td class="s0">gpt-4-0125-preview</td><td class="s0">$0.01 / 1K input tokens + $0.03 / 1K output tokens</td><td class="s0">N/A</td><td class="s0">N/A</td><td class="s0">N/A</td></tr></tbody></table></div>