Update README.md

README.md

@@ -1,3 +1,122 @@
 ---
 license: apache-2.0
+language:
+- en
+pipeline_tag: conversational
+tags:
+- not-for-all-audiences
 ---
+
+# Limamono-7B (Mistral) v0.3
+This is a very early version of a strongly NSFW roleplaying model with _extremely limited amounts_
+of almost entirely synthetic data of hopefully higher quality than typical human conversations.
+
+Limamono tries to address the main issues and limitations of the previously released [LimaRP](https://huggingface.co/datasets/lemonilia/LimaRP)
+and is composed of extensively modified conversations written with the help of base [Yi-34](https://huggingface.co/01-ai/Yi-34B)
+model by 01.ai.
+
+A defining characteristic of Limamono is _mind reading_. Characters may (not necessarily always)
+include thoughts in a seamless fashion inside their utterances.
+
+The prose style of this model is a somewhat extended book/novel format (further detailed below).
+Other formats are not supported and may conflict with the special features of this model.
+
+**Note**: there is currently no plan to release the dataset.
+
+## Prompt format
+Limamono uses the [extended Alpaca format](https://github.com/tatsu-lab/stanford_alpaca),
+with `### Input:` immediately preceding user inputs and `### Response:` immediately preceding
+model outputs. It's been trained with a fixed "trigger phrase" similar to that of the original
+Alpaca before the `### Instruction` sequence, following a similar template.
+
+```
+Below is an instruction that describes background information for a story-rich chat. Write an appropriate response for both the instruction and user input.
+
+### Instruction:
+{{char}}
+{{description}}
+
+Scenario: {{scenario}}
+
+### Response:
+{{char}}: [utterance]
+
+### Input:
+{{user}}: [utterance]
+
+### Response:
+{{char}}: [utterance]
+
+[etc...]
+```
+
+More in detail, the instruction should _preferably_ include a moderately long (a few hundred tokens
+long) character description made in the style of the various fandom wikis on the Internet, with the
+character name as the first line.
+
+### Message length control
+Inspired by the previously named "Roleplay" preset in SillyTavern, like with LimaRP it is possible to
+append a length modifier to the instruction sequences in this way:
+
+```
+### Response: (length = medium)
+{{char}}: {utterance}
+
+### Input: (length = tiny)
+{{user}}: {utterance}
+```
+
+This has an effect on bot responses, but as of now it might not always reliably work. The lengths
+using during training are: `micro`, `tiny`, `short`, `medium`, `long`, `massive`, `huge`.
+
+- It is recommended not to use a length modifier at all for character responses, except as a
+  _temporary measure_ for increasing message length.
+- It is suggested to add `(length = tiny)` to the `### Input:` sequence, on the other hand.
+
+## Prose style
+Only The Novel/Forum RP style is supported.
+
+### Style details
+- Narration does not have any delimiter.
+  - `Jessica looked at Mark with disdain.`
+- Dialogues wrapped with with ASCII double quotation marks. Fancy quotes are not supported.
+  - `"I say this."`
+- Onomatopoeias are wrapped with with asterisks.
+  - `*thud*`
+- Character thoughts are wrapped with underscores. **This may often occur with Limamono.**
+  - `_What is he doing?_`
+- Non-dialogue quotes are wrapped with two apostrophes on each side. This avoids conflicts with quotation marks in SillyTavern.
+  - `''The Jungle Book''`
+- Punctuation has been normalized and tries to follow standard convensions for book/novel writing.
+
+## Example
+This is how a typical RP chat may take place with this model. Note the presence of
+character thoughts.
+
+![example](https://files.catbox.moe/ch2bo2.png)
+
+## Training procedure
+[Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) was used for training
+on one NVidia RTX3090.
+
+### Training hyperparameters
+- load_in_8bit: true
+- adapter: lora
+- sequence_len: 4096
+- sample_packing: false
+- pad_to_sequence_len: true
+- lora_r: 8
+- lora_alpha: 16
+- lora_dropout: 0.5
+- gradient_accumulation_steps: 1
+- micro_batch_size: 1
+- num_epochs: 3
+- optimizer: adamw_torch
+- lr_scheduler: constant
+- learning_rate: 0.00018
+- weight_decay: 0.1
+- train_on_inputs: false
+- group_by_length: false
+- bf16: true
+- fp16: false
+- tf32: true