> Nobody fully understands these networks. Not even the people who build them. The "experts" admit this.

That's repeated a lot but it's not entirely accurate. No one can explain how an LLM gives the answers it does (not even the LLM). LLMs have a vast search space of tokens and use probabilities to make their responses non-deterministic. But the people who build and train LLMs do know how they work -- obviously since quite a few people know how to make one. By analogy, if I give a pile of Legos to a six-year-old I don't know what they will make, though I do know the constraints and limits (imposed by how Legos work and what was in the pile). It's not correct to say "I don't understand how Legos work" when I really mean "I can't predict what a six-year-old will make from a pile of Legos."

> You misunderstand how trivial it is to use this technology and figure out the limitations ... Hallucinations aren't technical level concepts.

I get that. But when I have discussed ChatGPT hallucinations, with examples from my own chats, I'm surprised when people don't even recognize the hallucinations until I point them out. Anecdotally people seem to defend the "AI" by accusing me of misleading it, or giving unclear information. They have anthropomorphized and then want to impose human notions of fairness, give the "AI" the benefit of the doubt even when they know that a person would have not made the mistake, or answered "I don't know" rather than confidently make stuff up like ChatGPT will. I think people don't believe computers can lie or make a mistake -- they imagine an intelligence like Mr. Spock at the other end, not a stochastic parrot.

I got ChatGPT to tell me -- in its confident and authoritative tone -- that no even number is also evenly divisible by 3. When I gave it contradictory examples -- 6, 12, 24 -- it then apologized but maintained that no even number was divisible by both 3 and 5 (um, 30, 60, 90...). I was just trying to get it to solve FizzBuzz with some variations. I could feed my younger children that same misinformation and they wouldn't question it. I could tell my parents that their Alexa listens to everything they say and records it forever on a big disk on a satellite in orbit, and they would believe me. Elon Musk can tell the world Teslas can drive from SF to New York without human intervention and get a whole TEDTalk audience and media "experts" to believe him. P.T. Barnum had some quips about that tendency.

> Find one guy on this entire thread who is honestly fooled by chatGPT and I'll concede to your argument. If you can't find even one guy... Then if you're rational you'll see that my argument is true: nearly no one is fooled by chatGPT.

That's not proof of anything. Maybe no one will chime in on this thread but you can easily find posts and comments on HN with people claiming LLMs are sentient. A Google researcher said that publicly (he got fired), and much discussion took place here. If I was more interested in the topic I could poll people, but I'm pretty sure I would find that quite a lot of people think current "AI" (LLMs like ChatGPT) are sentient, or they will be in the next couple of years. And then I could ask them what "sentient" means and never stop face-palming.