README initial

README.md

@@ -1,3 +1,36 @@
 ---
 license: mit
 ---
+
+# magic-the-gathering-flan-t5-xl
+
+A text generation model finetuned on Magic: The Gathering cards up to the Phyrexia: All Will Be One set (February 10, 2023). The base flan-t5-xl was finetuned for 1 "epoch" of card data using the LoRA technique in the PEFT library; this repo contains about 9M hyperparameters for the LoRA adapters.
+
+This model has strong support for out-of-domain inputs, such as generating a card with the name "San Francisco" (all generations will be Lands) and
+
+## Usage
+
+It is very strongly recommended to use this Colab Notebook to generate from the model as it requires some preprocessing and postprocessing both to get the inputs/outputs into the correct format and to extract the generated results.
+
+The tokenizer assumes a `\n` token is added to the default T5 tokenizer, as that is necessary for proper detokenization and the model was trained in that way. You can do this by adding `tokenizer.add_special_tokens({"additional_special_tokens": [AddedToken("\n")]})`
+
+## Training Techniques
+
+The model was also trained with two techniques to increase diversity and coherency of the output: hierarchal sampling and subset sampling.
+
+### Hierarchal Sampling
+
+There are many, more Creature cards than any other types of cards in Magic, therefore any model trained on a straight corpus will be biased toward Creatures. To work around this, a train-time data processor is used to
+
+Two caveats here: a) we can no longer guarantee the model will ever see all input data and b) it will likely see redundant cards from underepresented groups and thus risks memorization. The latter one can be fixed with...
+
+### Subset Sampling
+
+Also during train-time, the model will receive a random subset of the fields in the input card (including receiving zero information, to generate a card from scratch). The approach also models how users would use the model in practice. This technique makes it extremely unlikely for the model to see the same inputs twice, even if it trying to predict the same card multiple times. It also encourages the model to learn from near-infinite combinations of semantic inputs, which works well for T5's encoder-decoder structure.
+
+This technique also creates an intentional data leakage between input/output, which is desirable for this use case to ensure selected inputs are present in the output.
+
+## Notes
+
+- The model is still very bad at generating cards that follow the Magic "color pie" like other similar models
+- The card generations remains coherent even at high temperatures (1.5) which is new.