xzuyn/GPT2-RPGPT-8.48M

Latest Version: 111,577 / 111,577 Steps (Epoch 1).

• 28,563,712 / 28,563,712 tokens seen (Epoch 1).
• 0 / 28,563,712 tokens seen (Epoch 2).
• 0 / 28,563,712 tokens seen (Epoch 3).

Model Info:

• Trained from scratch.
• 8.48M parameters.
• 256 context length.
• Test model. Likely needs at least 512 context to function "properly".
• Trained with a dataset that overlaps by a quarter of the context length (Shifts by 64 tokens for each subset).

Format:

<|characters|>
Nancy (Oliver Twist): Female, early 20s, ESFP, Cockney accent. Loyal...
Mr. Edward Hyde (Dr. Jekyll and Mr. Hyde): Male, late 30s, ESTP...
<|scenario|>
In an alternate Victorian London where the city's poor and downtrodden...
<|response|>
Nancy: *gently brushes her fingers across the worn book spine, before suddenly stopping as she feels another hand...
Mr. Edward Hyde: *glances at Nancy with a sinister grin, slowly pulling his hand back* No need to apologize, miss...

Example Output:

Step 111,577. Input <|characters|> as a prompt, set max tokens to 256, amount to generate to 253. This generated up to just our circumstances before us. Then I set amount to generate to 128 to keep half of the text in context. This generated up to A wise suggestion,. I then lowered the amount to generate to 64. That generated up to the ending know of our current situation?.

<|characters|>
Mrs. Samsa (The Metamorphosis): Female, middle-aged, ISFJ, German accent, compassionate mother struggling to cope with her son's transformation, and eventually succumbs to the family's financial and emotional burdens.

<|scenario|>
In a twist of fate, Mrs. Samsa finds herself transported back in time to time and space. Evangelist, who is on an isolated haven where he encounters Mrs. Samsa, by a different tale. Mrs. Samsa, still burdened by the weight of his past actions, must confront the difficult path ahead.
Through their conversations, they find common ground in their own worlds, allowing them to continue seeking wisdom from each other and finding solace in one another's words. The dialogue between these two characters will offer insight into each other's worlds as well as how their experiences have shaped them in this whimsical world.

<|response|>
Mrs. Samsa: *approaches the peculiar sights around her, eyes widening in surprise* Oh dear, I couldn't help but notice you not! I've never seen my fair life, but I'm starting to see my son. Are you here in this peculiar place?
Evangelist: *smiles warmly at Mrs. Samsa* Yes, we are indeed more than just our circumstances before us. And it is your place of wisdom and understanding. *opens the book, his eyes sparkling with excitement*
Mrs. Samsa: *slowly opens a small book of the book* I must confess, Evangelist, I've never had a different view of this place. But it feels like this before our worlds find such things that we've discovered.
Evangelist: *nods thoughtfully* You possess great wisdom, Mrs. Samsa. It seems we are both searching for a way to escape this peculiar library. Perhaps that is a sign of my spiritual journey towards you.
Mrs. Samsa: *eyes widen in curiosity* A wise suggestion, Candide. I can't help but feel a sense of serenity amidst my own life.
Evangelist: *smiles warmly* Of course, Mrs. Samsa. The path to enlightenment is filled with joy and understanding. Now, tell me more about this ancient book. What do you need to know of our current situation?

Config:

Learning rate may have been too high, not sure. Average loss at step 111,557 had an averge loss of 2.1.

batch_size: 1
dropout: 0
learning_rate: 0.0001
max_length: 256
n_embed: 256
n_head: 8
n_layer: 8
vocab_size: 8192