> Could you detail what the results of that problematic behavior are?

I'll contrast two sampling approaches (ignoring prompting and whatnot since that's orthogonal and can be applied to either): (1) repeat till the answer adheres to a grammar, and (2) filter the set of possible next tokens to those which could adhere to that grammar.

The former preserves the same relative probabilities in valid answers that the base model would have, whereas the latter as an unknown, not intuitively explainable, and wildly differing distribution.

Anecdotally, one way in which that manifests is that when the model inevitably makes a grammar-admissible mistake on some token, the base model is more likely to also make a mistake in the grammar than it otherwise would have. Sampling strategy (1) throws that mistake away, where sampling strategy (2) silently forces the rest of the answer to comply, yielding an incorrect but "valid" answer. The long-string-ellipses problem I described falls into that class of problems.

The specific behaviors are hard to qualitatively describe in full generality though because of the huge number of ways in which a "wrong" probability distribution can be wrong.

> Do you suspect structured generation performs worse than unstructured?

Yes, often, not always. E.g, if your schema is enum{Red,Blue,Green}, you can prove that the two distributions are in fact equal, so structured generation would be strictly better because it's cheaper. For specialized problems, structured generation is another meta-parameter, and despite having no intuitive explanation for _why_ it performs better, if it accidentally performs better for your particular problem then that's a huge win. Even if it increases the error rate, it's also cheaper than the alternatives (and most reasonable applications of an LLM assume a nontrivial error rate anyway, so that's not necessarily a huge cost), so it might be "better" for an application despite lower quality results.

For complicated schemas though, I'd definitely at least want to measure the difference. Anecdotally, structured generation as a sampling procedure performs worse for me on complicated problems than unstructured generation.

> The strategies I've seen for dealing with structured generation seem to be ensuring that the structured is represented in the prompt, so the model isn't squishing anything.

"Squishing" was a bit of a colloquialism. For the vast majority of problems I've seen, fitting the grammar into the prompt, choosing a very very simple grammar, and filtering non-confirming responses is a pretty good approach. Your success rate is decently high (i.e., not much more expensive than sampling-based approaches), you get something sufficiently machine-readable to fit into your pipeline, and you have a sampling distribution matching the underlying LLM. For sampling-based approaches though, you skew the result distribution in the way described above, which I called "squishing".

> Part of structured generation is specifying exactly the range you want for the field, so I'm not sure how this issue arises. With structured generation you force the LLM to have at least some number of characters, so this seems like an argument in favor of structured generation rather than against. If unstructured LLMs want to "close the field", then structured can force it to remain open.

If you can enumerate all the failure modes (or likely failure modes), absolutely. My biggest counter-arguments are:

1. That's hard to enumerate in general (hence why we have LLMs instead of grammar-rule-machines).

2. Even for the ellipses example, if you want to use one of those structured json libraries, how hard is it to require json and also require that "certain" strings (over X length, with certain corresponding keys, ...) can't be ended as `..."`? I haven't done it, but I have enough programming experience to be pretty sure it'll be a pain in the ass unless you want to fork the library for your use-case or re-write most of its json grammar.

3. Suppose you get that constraint (running example, certain strings can't end via `..."`) into the grammar, what exactly does that do? Those algorithms are greedy, and when the model first inserts a mistaken period, it's already made a mistake. Periods are allowed in strings though, so the grammar continues. The model, having differed from the text it's transcribing and inserted a period, inserts another period because it's obviously creating an ellipses. This is still valid in the grammar, so text generation continues. If your grammar just banned ellipses, you'd get some other nonsense character or an end-of-string-quotation at this point, and if you banned string-ending-ellipses you'd almost certainly get another period, at least one nonsense character, and then an end-of-string-quotation. Despite the fact that you banned the bad behavior, in all the cases you would have seen it without structured generation, you still get broken outputs with structured generation and don't know that they're broken. Contrast that with letting the string complete and re-trying if it ends as `..."`. Every time the model mistakenly adds a period trying to end the string, that is explicitly caught and tracked. It's still "structured", but the result distribution is different and better.