More or less. I have tools I wrote that takes sampling settings and a string and dispatches it across a cluster of machines. Via cluster-ssh I see the each of them expanding the text in a different random sampling.

This is custom stuff I've wrote that is tied to my environment. If you'd just like to play around, I can recommend https://bellard.org/nncp/gpt2tc.html in text generation mode as being extremely easy to get going. (the default model however is pretty small and dumb). The paradigm you're thinking about is exactly what you get from gpt2tc.

At any point I can abort a job, tweak the sampling settings, or the text I'm expanding. E.g. one operation is that if I see one sample seems to be on a good path but has made an error, I'll abort it and restart all of them from a fixed version of that sample.

Often I'll end up with my prompts in text files because they get a bit long at times, also escaping quotes and linebreaks on the commandline can be a pita.

I have some aspirations of integrating this into a text editor, so that as I type future text is just appearing ahead of me and I can just hit a cursor to accept parts of it. But in my experience GPT2 isn't good enough where looking only at one continuation is enough or where I don't get a lot of advantage in having it work from modified text.

GPT2 has preconceived notions about what kind of text you're writing based on the words you use. So it can be useful to alter your input text to replace persons/places/things names with different ones that get in into the right context and then back substitute them.

To give a concrete example, if I wanted GPT2 to show me example bio blurbs for my partner (always a pain to write but easier if someone generates examples), it works better if I change her name--Kat-- because it either turns her into a man or it resists talking about her being a lawyer and a board member and instead makes her into an artist or a dancer.

One thing to watch out for is that when GPT2 makes a benign error, like switching the gender of a pronoun mid-stream it often trashes the quality of the later output in unexpected ways (like causing it to output nonsense). Changing my SO's name to something it's not unsure about saves me time having to abort completions that have gone off the rails.

Perhaps I shouldn't use a gendered example. GPT2 isn't sexist its everything-ist. Every word has 1001 hidden meanings that subtly bias its behaviour, many of the biases are actually the point- they exist in the world and they're what makes the output useful-- others are weird and unexpected and are just a training/corpus artifact. Good use requires a degree of anticipating and exploiting these biases.

I even have to change my own name, because GPT2 knows very well that "Greg Maxwell" has something to do with Bitcoin and it readily lapses into Bitcoin conspiracy theories if my name is used. ["Gessh, even the machines are harassing me!"]

Another class of biases is language that trips it into fantasy land or 'silly' writing. Any word that is commonly used in writing for children is at risk. If there is another common word that only adults use it might be a better choice if you want serious text, or you setup a context that makes the meaning more clear.

It looks like for GPT3 you don't have to use subtle hints as much. E.g. that you can just tell it more explicitly what kind of thing you're doing and won't get tripped up as much by spurious correlations.

OTOH, since it doesn't look like they're going to release the GPT3 model it's likely that I'll never get an opportunity to use it in my workflow.