language:
- en
license: creativeml-openrail-m
thumbnail: https://huggingface.co/volrath50/fantasy-card-diffusion/collage.jpg
tags:
- stable-diffusion
- text-to-image
- image-to-image
#fantasy-card-diffusion
A comprehensive Stable Diffusion model for generating fantasy trading card style art, trained on all currently available Magic: the Gathering card art (~35k unique pieces of art). Has a strong understanding of MTG Artists, planes, sets, colors, card types, creature types and much more.
For a guide on using the model, scroll down below
Features
- Incorporate the styles of artists you know and love from Magic: Gathering
- Produce art that looks like it is from a given MTG plane, set or year
- Create fantasy creatures in the style as they exist in Magic the Gathering
- Draw fantasy creature types that are unique to MtG (like Eldrazi)
- Use well known MTG characters (such as the planeswalkers)
- Draw real-world or non-MtG characters, in the style of MTG art
- Mix and match all of the above
Using the Model
The model was trained on MtG card information, not art descriptions. This has the effect of preserving most non-MtG learning intact, allowing you to mix MtG card terms with an art description for great customization.
Each card was trained with card information pulled from Scryfall in the following format:
MTG card art, [Card Name], by [Artist], [year], [colors (words)], [colors (letters)], [card type], [rarity], [set name], [set code], [plane], [set type*], [watermark*], [mana cost], [security stamp*], [power/toughness*], [keywords*], [promo type*], [story spotlight*]
A few examples of actual card data in this format: MTG card art, Ayula, Queen Among Bears, by Jesper Ejsing, 2019, Green, G, Legendary Creature - Bear, rare, Modern Horizons, mh1, draft_innovation, 1G, None, 2/2, Fight, MTG card art, Force of Will, by Terese Nielsen, 1996, Blue, U, Instant, uncommon, Alliances, all, Dominaria, Terisiare, Ice Age, expansion, 3UU,
Training and dataset
Training was done on a dataset consisting of cropped, 512x512 versions of the art for every MTG card, each of which was tagged using a custom python script, from data pulled from Scryfall. Training was done with the Dreambooth extension for Automatic1111's wonderful UI, to 130,000 steps.
The result is a comprehensive model that has a good understanding of MTG artists, sets, planes, card types, creature types, years, colors, and more. If you had ever wondered what a Merfolk, drawn by Ron Spencer, would have looked like on Tarkir, as part of the Mardu clan, with dash, haste, and trample - this model can deliver what you want.
Because the training data is literally the art from every MTG card, combined with the information for the associated card (about ~35,000 unique pieces of art and text), I won't be releasing the training data, out of concerns that would be violating WotC's IP. I have, however, included the python script that I used to generate the training data set, which should get you uncropped images and identical text (or near identical) text files, with used with the "unique artwork" json from https://scryfall.com/docs/api/bulk-data
My script could probably be written much better. I have never studied python, and, up until I wanted to make this model, had not done any programming at since about 2000-2001, when I was a teenager and liked Perl. I managed to hack it together in a weekend using 20+ year old memories of Perl, liberal use of Github Copilot, and a lot of googling.
Cropping was done with ImageMagick (see below, under issues).
Issues
This was intended to be a second test run on the full data set (the first did not go well), so some corners were cut for the purpose of starting my "testing." The model turned out far better than I had expected, so I've decided to release it as is, and hope other people enjoy it as much as I have. But there are some issues that I am aware of and intend to work on fixing for future releases:
- Cropping -- MTG art is rectangular. I initially tried to use a trainer that could handle different aspect ratios, but after a couple failed tries, I just did a quick mass cropping job with ImageMagick, resizing and cropping everything to 512x512, so I could get training running. I forget what exactly I did, but it appears it focused on the left side of the card, universally cutting off the right side. You'll see this in generations, that tend to have everything on the right, as that's the image the training robot saw. I will fix this in future releases.
- Planes -- Plane information was only added around step 70,000, so it may be less trained than other information - basically, I wanted a way to group sets together by plane, as I was finding how well it knew the look of a set depended on whether WotC had incorporated the name of the plane into the set itself - ie: using "Theros" would only get you "Theros" and "Theros: Beyond Death" and not "Born of the Gods" or "Journey into Nyx"
- Unique Characters -- How well it knows a unique character largely depends on how many cards they show up on, and how well the AI was able to isolate the character in question -- Most Planeswalkers are fairly well known, even some that have very few cards. (It knows who Aminatou is, despite her having a single card, and even managed to separate her from Seb McKinnon's awesome artwork) -- Some other unique characters, it only has a general sense of what they look like - Emrakul, for instance, comes out more as purple version of the flying spaghetti monster, and some creature types - (like slivers), it still hasn't quite figured them out
- Random issues -- Some artists use special characters in their name. I tried to take away all accents, but I missed at least one, Tom Wänerstrand, who is trained as Tom Wänerstrand, with the umlaut -- Greg Rutkowski: Not an issue, but the poster boy for AI art, Greg Rutkowski, is an MTG artist. He uses the Polish form of his name on MTG cards, Grzegorz Rutkowski, and that is what this model was trained with. So you'll get different results using "by Greg Rutkowski" vs "by Grzegorz Rutkowski"