I want to preface this by saying I'm not trying to hate on the author. Good on them for applying a new technology to try to solve an annoying part of their work. I hope they continue to be creative and fearless in their approach to problem solving. The comments that follow are born more out of my increasing frustration with the ML and ML-adjacent communities that seem to have failed to learn anything about the dangers of hype and over-promising from the 1980s (and really just human history in general).

While it's an undeniable achievement that these kinds of models can generate syntactically correct code or even the correct solution some of the time. I find it hard to get excited about them for anything but boilerplate (and even they I think they pose the greatest risk, lulling one into a false sense of security until you get bitten).

If we take formal methods as the way to ensure the code we've written provably does what we say it does, we had better damn well make sure that what we've said it does is what we want it to do. In other words, I feel like this is offloading the most important part of the problem, and the one that can't be checked automatically onto a very iffy situation.

We should really stop to consider what these kinds of models are. Pattern recognizers, not thinking machines. See the author's quote about this theorem:

Let G be a Lie group. Then the tangent space at the identity is a Lie algebra.

which is translated into the theorem:

theorem tangent_space_at_identity_is_lie_algebra (G : Type*) [group G]
[topological_space G] [topological_group G] [charted_space G] [smooth_manifold G] :
lie_algebra (tangent_space_at_identity G) :=

Their quote

For some completely unclear reason, the app decides to consider a Lie algebra over \mathbb{R} instead of my explicit instruction to use a field k, and that’s even before addressing the fact that tangent_space_at_identity is not a mathlib definition. I tried a few different responses to the app’s initial attempt, and couldn’t get it to do anything helpful.

They try to fix this by "prompt engineering":

You must specify a field for your Lie algebra as in lie_algebra k M

which leads to the change:

lie_algebra R (tangent_space_at_identity G) :=

This is exactly the kind of boilerplate we'd want to automate with this. The specification of a Lie group seems onerous here, but it's also exactly the kind of mistake that makes using this kind of mistake that makes relying on these systems so scary. Here it's a syntax pattern, and we have to end up proving the correctness, so maybe we wouldn't notice the mistake here. It's almost certainly going to be missed often in (choose your favorite dynamically typed interpreted language that let's you find all this fun stuff out at runtime). There is no understanding of what we do and do not know. What we've proved or have yet to prove be damned.

I guess my point hinges completely on what you consider "knowing" to be. It's kind of funny. Nobody would ever say that a compiler knows how to transform one language to another. The personification of ML really irks me. I try very hard to avoid it, but it's easy to slip.

I don't want to say we should throw the baby out with the bathwater. I do think that there are uses for these things. The recognition of the pattern for the syntax to specify the characteristics of a Lie group would make a great first pass at a snippet template. (This is ignoring the question of why isn't Lie_group a definition that just pulls in the necessary requirements on G rather than needing to specify them each time. If one of those is not really a requirement for Lie groups, forgive me, my exposure to Lie theory is very limited.)

I would be much more interested in a tool that allows me to identify patterns in the code with a summary printout or something like the likelihood of a given pattern in the neighborhood of my cursor. The generation part is just not good enough to trust IMO.

Also, I really hope we see some sort of backlash for MS just deciding to just ignore everyone's' licenses.