I think there's a very narrow window where I'm getting useful results out of it, enough that I'm not completely writing it off as a way to help out with some productive work. I do expect I'll make use of it occasionally while programming in the future. But all of the times I've used it, it hasn't been an extremely clear productivity boost (certainly not at the x2/x10 level), and I do need to babysit it way more than people are suggesting.

I go back and forth in my opinion on it. There are some topics where I get useful sessions, there are some topics where it's a mess. The problem is I don't know in advance which are which. It seems to be most useful in scenarios where I either need some extra motivation to get over the initial hump of starting a task (making an task feel more approachable is I think a great use-case), or where I know just enough about a task to know if it's generating nonsense, but not enough to be able to knock out the same results in less time myself.

The other problem is that in areas where I'm not immediately familiar with what's going on, I sometimes have to spend a lot of time debugging its code. And reading and debugging other people's code is (to me) a lot harder than writing code myself. So there's an inherent slowdown there where I have to check its work as if I am reading a junior programmer's code and checking it for errors. Code reviews are annoying and take a lot of mental effort to do well. You can ask GPT-4 to explain itself, but there's no way to know without actually stepping through the code if its explanation is correct.

I also worry that it's railroading me a bit. I had it prototype an algorithm for me and debugged it and helped it expand the algorithm and get it working with the results I wanted. Only afterwards did I realize there was a much simpler way to get the result I wanted without invoking the algorithm at all. If I wasn't in a conversational loop with an AI focused on getting its code to work, I might have noticed that earlier. But my brain was only focused on debugging the code for that task, it wasn't focused on going back and rethinking the algorithm and cutting out the useless bits.

BUT... sometimes it seems helpful with prototyping? So it's not all bad. I don't necessarily want to dismiss it entirely.

My take is potentially useful, and for some people who are particularly suited for this kind of rubber-duck peer-coding style of work, maybe a big increase in productivity? But certainly not universally useful for every programmer, and I can still see the seams. GPT-4 is significantly better than GPT-3, but it's not as good as people have hyped it to me, and I think people are vastly over-estimating its capabilities. I went in skeptical, but even with that attitude from what people had told me beforehand I still expected GPT-4 to be better than it is.

Am I worried this is going to steal my job? No. That's honestly a kind of laughable idea to me after seeing the kind of code it writes and how unreliable it is.

----

It's hard to put this into words, but it also kind of hints to me at a much better system, and I can see problems here around the auto-regressive nature of the LLM and the need for alignment training and around the whole structure of the thing and the slowness of generating text over and over, and the sort of forced linearity of the chat session that makes me think that maybe GPT-4 is just kind of hinting at a debugging/prototyping methodology that would be a massive improvement to productivity but that doesn't necessarily involve LLMs or involves them to a much lesser degree. It's in some ways a very clumsy/brutalist approach to simulating a feedback repl and accessing documentation, and because it's clumsy/brutalist it over-emphasizes the natural language aspect. And I'm curious to see if future projects can identify what specifically is helpful here and deliver that without all of the inherent downsides of needing to use a conversational model.

That's all ignoring the privacy/data concerns, but they're a separate problem.