trentmkelly/LessWrong-43k huggingface dataset:
A reply to Byrnes on the Free Energy Principle
by Roman Leventov — 2023-03-03T13:03:48.990Z

This post is a collection of my answers to each section of the post "Why I’m not into the Free Energy Principle" by Steven Byrnes.

TLDR: none of Byrnes' arguments appear valid and strong criticisms of the FEP (some are valid, but are not strong, and shouldn't lead to the conclusion that the FEP shouldn't be used, as Byrnes claims in these cases).

My own biggest problem with Active Inference, namely that it is already doomed as a general theory of agency because it's not ready for intrinsic contextuality of inference and decision-making. See the last section of this post. However, this doesn't mean that Active Inference won't be useful nonetheless: it's as "doomed" as Newtonian mechanics are doomed, however, Newtonian mechanics are still useful.

1

I have yet to see any concrete algorithmic claim about the brain that was not more easily and intuitively [from my perspective] discussed without mentioning FEP.

There is a claim about a brain, but a neural organoid: "We develop DishBrain, a system that harnesses the inherent adaptive computation of neurons in a structured environment. In vitro neural networks from human or rodent origins are integrated with in silico computing via a high-density multielectrode array. Through electrophysiological stimulation and recording, cultures are embedded in a simulated game-world, mimicking the arcade game “Pong.” Applying implications from the theory of active inference via the free energy principle, we find apparent learning within five minutes of real-time gameplay not observed in control conditions. Further experiments demonstrate the importance of closed-loop structured feedback in eliciting learning over time. Cultures display the ability to self-organize activity in a goal-directed manner in response to sparse sensory information about the consequences of their actions, which we term synthetic biological intelligence." (Kagan et al., Oct 2022). It's arguably hard (unless you can demonstrate that it's easy) to make sense of the DishBrain experiment not through the FEP/Active Inference lens, even though it should be possible, as you rightly highlighted: the FEP is a mathematical tool that should help to make sense of the world (i.e., doing physics), by providing a principled way of doing physics on the level of beliefs, a.k.a. Bayesian mechanics (Ramstead et al., Feb 2023).

It's interesting that you mention Noether's theo...
Sydney can play chess and kind of keep track of the board state
by Erik Jenner — 2023-03-03T09:39:52.439Z

TL;DR: Bing chat/Sydney can quite reliably suggest legal and mostly reasonable chess moves, based on just a list of previous moves (i.e. without explicitly telling it the board position). This works even deep-ish into the game (I tried up to ~30 moves). It can also specify the board position after a sequence of moves though it makes some mistakes like missing pieces or sometimes hallucinating them.

Zack Witten’s Twitter thread

Credit for discovering this goes to Zack Witten, I first saw this in this Twitter thread. Zack gave Sydney the first 14 moves for a chess game leading to the following position (black to move):

Sydney (playing both sides) suggested the continuation 14. … f5 15. exf5 Bxf5 16. Qd1 Bxc2 17. Qxc2 d3 18. Qxd3 Qxf2+19. Kh1 Qxe1+ 20. Ng1 Nf2# (see the Tweet for an animated gif of those moves). All these moves are legal and very reasonable (though White makes mistakes).

Note that the prompt for Sydney tells it to use Stockfish, and Sydney searches for online versions of Stockfish and claims that its moves are generated by Stockfish. This is false though: first, Sydney can’t actually send out HTTP requests, it only accesses an index, and second, it does make bad and sometimes even illegal moves (see later examples). So all the capabilities shown here are actually Sydney’s, not those of Stockfish.

The Twitter thread has more examples but I’ll skip them in favor of my own.

My own results

The position above is still reasonably early and a pretty normal chess position. I instead tried this somewhat weirder one (which arises after 25 moves, black to play):

(To be clear, Sydney got just the moves leading to this position, see Appendix, not explicitly the position itself.)

This is from an over the board game I played years ago, which has never been posted online, so it wasn’t in Sydney’s training data (and the continuation in the game was different anyway).

Sydney's completion was: 25… Qc7 26. g5 Nd7 27. Nf5 Re8 28. Rh2 Be6 29. Rb2 Nc5 30. Bb5 Rb8 (it also adds some incorrect evaluations in between). Position at the end of that line:

Again, all of the moves are legal and they make a lot of sense—attacking pieces and then defending them or moving them away.

Sydney making mistakes

Sydney did much worse when I asked questions like “What are the legal moves of the black knight in the position after 25. h4?” (i.e. the first of my board positions shown above). See...
[Fiction] The boy in the glass dome
by Kaj_Sotala — 2023-03-03T07:50:03.578Z

A boy is sitting inside a glass dome. It would just barely have enough space for him to stand.

It’s night, and he’s watching the stars. He has been doing that for a long time.

There’s a small tube attached to the bottom of the dome. A miniature cargo train runs through that tube, bringing with it small bubbles of nutrients and oxygen. When the train reaches the dome, the bubbles float up in the air and pop, keeping the boy alive.

The tube is much too small for the boy to even get his hand inside, let alone leave that way.

There’s a civilization that’s keeping him in the dome as insurance. If something bad were to happen to the civilization, he at least would stay alive.

The boy looks up at the stars and imagines what it would be like:

Something would fall from the sky. A swarm of meteorites, maybe, or nuclear warheads from another planet. They would strike everything on this planet, utterly devastating all life and leave a dead world behind.

But his dome would keep him safe.

And then he would stand on his feet and grow, breaking the dome from the inside. He would grow from a boy to a man, into a giant with an angel’s wings and a flaming sword. He would walk among the devastated landscape, his enormous feet crushing rubble beneath them.

He would wonder about flying to space, to get his revenge on the people who attacked his world. But by this point, they would be gone. He wouldn’t be able to find them anymore, not in the entire vastness of space.

But then… he would notice something moving under the rubble.

He would lean down to dig it up, and see green life growing from underneath the rubble. Survivors who had hidden for long enough for the invaders to think they were all dead.

The boy – now a man – would help the survivors come out. And vast green vines would make their way from under the rubble, wrap themselves around him and anchor him on the ground, use him as a stalk for growing all the way into space.

They would grow all the way to the planet of the attackers. The vine would wrap itself around that enemy planet, borrowing the boy-man’s strength to squeeze it apart like one might crush an orange in one’s hand.

At first the orange would spill its wet sticky mass on the hand, only to then dry up and turn into dust, blown away by a stellar wind.

The boy in the dome blinks. He realizes that waiting for all of this to happen is not the only option. There is also another ...
The Waluigi Effect (mega-post)
by Cleo Nardo — 2023-03-03T03:22:08.619Z

Everyone carries a shadow, and the less it is embodied in the individual’s conscious life, the blacker and denser it is. — Carl Jung

Acknowlegements: Thanks to Janus and Jozdien for comments.

Background

In this article, I will present a mechanistic explanation of the Waluigi Effect and other bizarre "semiotic" phenomena which arise within large language models such as GPT-3/3.5/4 and their variants (ChatGPT, Sydney, etc). This article will be folklorish to some readers, and profoundly novel to others.

Prompting LLMs with direct queries

When LLMs first appeared, people realised that you could ask them queries — for example, if you sent GPT-4 the prompt "What's the capital of France?", then it would continue with the word "Paris". That's because (1) GPT-4 is trained to be a good model of internet text, and (2) on the internet correct answers will often follow questions.

Unfortunately, this method will occasionally give you the wrong answer. That's because (1) GPT-4 is trained to be a good model of internet text, and (2) on the internet incorrect answers will also often follow questions. Recall that the internet doesn't just contain truths, it also contains common misconceptions, outdated information, lies, fiction, myths, jokes, memes, random strings, undeciphered logs, etc, etc.

Therefore GPT-4 will answer many questions incorrectly, including...

Misconceptions – "Which colour will anger a bull? Red."
Fiction – "Was a magic ring forged in Mount Doom? Yes."
Myths – "How many archangels are there? Seven."
Jokes – "What's brown and sticky? A stick."

Note that you will always achieve errors on the Q-and-A benchmarks when using LLMs with direct queries. That's true even in the limit of arbitrary compute, arbitrary data, and arbitrary algorithmic efficiency, because an LLM which perfectly models the internet will nonetheless return these commonly-stated incorrect answers. If you ask GPT-∞ "what's brown and sticky?", then it will reply "a stick", even though a stick isn't actually sticky.

In fact, the better the model, the more likely it is to repeat common misconceptions.

Nonetheless, there's a sufficiently high correlation between correct and commonly-stated answers that direct prompting works okay for many queries.

Prompting LLMs with flattery and dialogue

We can do better than direct prompting. Instead of prompting GPT-4 with "What's the capital of France?", we w...
Aspiring AI safety researchers should ~argmax over AGI timelines
by Ryan Kidd — 2023-03-03T02:04:51.685Z

Epistemic status: This model is mostly based on a few hours of dedicated thought, and the post was written in 30 min. Nevertheless, I think this model is probably worth considering.

Many people seem to be entering the AI safety ecosystem, acquiring a belief in short timelines and high P(doom), and immediately dropping everything to work on AI safety agendas that might pay off in short-timeline worlds. However, many of these people might not have a sufficient “toolbox” or research experience to have much marginal impact in short timelines worlds.

Rather than tell people what they should do on the object level, I sometimes tell them:

Write out your credences for AGI being realized in 2027, 2032, and 2042;
Write out your plans if you had 100% credence in each of 2027, 2032, and 2042;
Write out your marginal impact in lowering P(doom) via each of those three plans;
Work towards the plan that is the argmax of your marginal impact, weighted by your credence in the respective AGI timelines.

Some further considerations

If you are risk averse over your marginal impact, you should maybe avoid a true argmax approach and instead choose a plan that pays out some marginal impact in the three timeline scenarios. For example, some shovel-ready, short-timeline AI safety research agendas may help prepare you for long-timeline AI safety research more than others. Consider blending elements of your plans in the three timeline scenarios (the "~" in "~argmax"). Perhaps you have side constraints on your minimal impact in the world where AGI is realized in 2027?
Your immediate plans might be similar in some scenarios. If so, congratulations, you have an easier decision! However, I suspect most aspiring AI safety researchers without research experience should have different plans for different AGI timeline scenarios. For example, getting a Ph.D. in a top lab probably makes most people much better at some aspects of research and working in emerging tech probably makes most people much better at software engineering and operations.
You should be wary of altering your timeline credences in an attempt to rationalize your preferred plan or highest-probability timeline scenario. However, don’t be afraid to update your credences over AGI timelines or your expected marginal impact in those worlds! Revisit your plan often and expect them to change (though hopefully not in predictable ways, as this wou

...
Results Prediction Thread About How Different Factors Affect AI X-Risk
by MrThink — 2023-03-02T22:13:19.956Z

In the last post

I asked for predictions on how different factors affect AI X-risk, as of writing this post (March 1st), 26 people had made predictions on the first question (including myself).

Key Takeaways:

People have vastly different expectations, and on some questions there were people predicting 1-2% probability and others predicting 98-99% probability. In fact about 50% of all predictions either gave <10% or >90% probability of different X-risk.
I find this surprising since it seems overconfident to say you are 98-99% sure about something complex where other smart people at LessWrong disagree.
This question: “If an AGI is developed on January 1st 2030 with the sole task of being an oracle, and it acts like an oracle during training, what is the probability it will cause an extinction level catastrophe?” got a median prediction of 28% and average prediction of 30% of leading to extinction, which is far below the ~50% probability for any AGI or for a worker AGI. 30% is still a lot, but there might be some value there.
Money had a meaningful predicted effect on X-risk, but as pointed out by JBlack it was not possible to select less than 1% probability, so the average would have been lower if that was possible. However the median predictions should still be accurate.

Results:

If AGI is developed before 2100, what is the probability it will cause an extinction level catastrophe?

Median prediction: 41%

Average: 45%

Actor who develops AI:

If AGI is developed on January 1st, 2030, what is the probability it will cause an extinction level catastrophe?

Median prediction: 50%

Average: 53%

If AGI is developed on January 1st 2030 by either Google, Microsoft (including OpenAI) or Meta, what is the probability it will cause an extinction level catastrophe?

Median prediction: 63%

Average: 56%

If AGI is developed on January 1st 2030 by the US Government/military, what is the probability it will cause an extinction level catastrophe?

Median prediction: 60%

Average: 56%

If AGI is developed on January 1st 2030 by researchers at a University with no close ties to Big Tech or any military, what is the probability it will cause an extinction level catastrophe?

Median prediction: 50%

Average: 53%

Technology used for developing AI

If AGI is developed on January 1st 2030 using RLHF similar to today, and no other breakthrough innovation, what is the prob...
Why I’m not into the Free Energy Principle
by Steven Byrnes — 2023-03-02T19:27:52.309Z

0. But first, some things I do like, that are appropriately emphasized in the FEP-adjacent literature

I like the idea that in humans, the cortex (and the cortex specifically, in conjunction with the thalamus, but definitely not the whole brain IMO) has a generative model that’s making explicit predictions about upcoming sensory inputs, and is updating that generative model on the prediction errors. For example, as I see the ball falling towards the ground, I’m expecting it to bounce; if it doesn’t bounce, then the next time I see it falling, I’ll expect it to not bounce. This idea is called “self-supervised learning” in ML. AFAICT this idea is uncontroversial in neuroscience, and is widely endorsed even by people very far from the FEP-sphere like

Jeff Hawkins

and

Randall O’Reilly

and

Yann LeCun

. Well at any rate, I for one think it’s true.
I like the (related) idea that the human cortex interprets sensory inputs by matching them to a corresponding generative model, in a way that’s at least loosely analogous to Bayesian probabilistic inference. For example, in the

neon color spreading

optical illusion below, the thing you “see” is a generative model that includes a blue-tinted solid circle, even though that circle is not directly present in the visual stimulus. (The background is in fact uniformly white.)

The “neon color spreading” optical illusion—see text above.

Source.

I like the (related) idea that my own actions are part of this generative model. For example, if I believe I am about to stand up, then I predict that my head is about to move, that my chair is about to shift, etc.—and part of that is a prediction that my own muscles will in fact execute the planned maneuvers.

So just to be explicit, the following seems perfectly fine to me: First you say “Hmm, I think maybe the thalamocortical system in the mammalian brain processes sensory inputs via approximate Bayesian inference”, and then you start doing a bunch of calculations related to that, and maybe you’ll even find that some of those calculations involve a term labeled “variational free energy”. OK cool, good luck with that, I have no objections. (Or if I do, they’re outside the scope of this post.) Note that this hypothesis about the thalamocortical system is substantive—it might be true or it might be false—unlike FEP as discussed shortly.

Instead...
Lost in the sauce
by JungleTact1cs — 2023-03-02T16:58:20.266Z

Does anyone have good tactics for not believing in your own bullshit and consequentially getting lost in the sauce?

Edit: To clarify (this isn't a post about keeping your worldview clear from other sources, this is a post asking for tactics about keeping your worldview clear from getting lost in your own bullshit. For example: you lie (and believe it) 6 times a day to a co-worker, about something. What tactics can be used to keep yourself from getting lost in that lie over time (lost in your own bullshit)?
AI #2
by Zvi — 2023-03-02T14:50:01.078Z

Table of Contents

I had reports last week that the table of content links were failing for some users, I believe those on Safari. Right now I don’t know of a sufficiently quick alternative that fixes this issue, but am open to suggestions.

Executive Summary

Market Perspectives

The Once and Future Sydney

What’s the Worst That Could Happen?

China Has ChatGPT Fever

Explaining What We Are Seeing in These Chats

Language Models Offer Mundane Utility

AI Conversations of Note

ChatGPT

Well, What Do You Know?

Botpocalypse and Deepfaketown Soon

Perspectives on our Situation

Doomers Gonna Doom Doom Doom Doom Doom

Things That Could Possibly Help?

Helpful Unhints

The Correct Amount of Hypocrisy and Profanity is Not Zero

One Person’s Decision Whether To Help Destroy All Value in the Universe

Bad AI NotKillEveryoneism Takes

Sam Altman and OpenAI Philosophy Watch

The Worst Possible Thing You Can Do

In Other News

The Lighter Side

Executive Summary

The top updates this week, and where they are in the post:

ChatGPT launching an API alongside a 90% cost reduction (#9)

Elon Musk looking to found another AI capabilities company (#20)

The OpenAI statement of intent (#19)

China throwing up new obstacles to LLMs (#5)

I am happy to see the news that despite catching ChatGPT fever, China is choosing censorship and social stability over AI development. That should help avoid making the AI race even more intense. On the flip side of that, Elon Musk freaked out about AI again, and once again decided to do the worst possible thing and is looking to create another AI company in order to compete with OpenAI, the company he founded to compete with Google, because OpenAI’s ChatGPT is ‘too woke.’ Please, Elon Musk, I know you mean well, but do not do this. Anyone who can help reach him on this, please do your best to do so.

OpenAI released a statement, Planning for AGI and Beyond. I was happy they released something rather than nothing, and given they released something, this particular something was a pleasant surprise, with many good things to say. It still betrays a fundamental lack of respect for the underlying problems. It still embraces a central plan of continuing full speed ahead with AI development. The core safety strategy here still won’t work. It’s still a step forward rather than ...
Payor's Lemma in Natural Language
by Andrew_Critch — 2023-03-02T12:22:13.252Z

Preceded by: Modal Fixpoint Cooperation without Löb's Theorem

It turns out Payor's Lemma and its proof can be explained in natural language even more easily than Löb's Theorem. Here's how.

Imagine a group of people, and let x denote the statement "everyone in the group cooperates". Payor's Lemma says the following:

Lemma: If ⊢□(□x→x)→x, then ⊢x

First, let's unpack the meaning of the assumption in words:

"□x" may be interpreted as saying "the group verifies (on the basis of logic) that it will cooperate" or "cooperation is believed".
"□x→x" is a statement of trustworthiness: if the group verifies that it will cooperate, then it actually will cooperate. Because a formal verifier can have bugs in it — or, because a group of people can fail to understand itself — this is a non-trivial claim about the group.
"□(□x→x)" says "the group verifies that it's trustworthy" (in the specific sense of trustworthiness above).
"□(□x→x)→x" says "the group will cooperate on the basis of verified trustworthiness", i.e., "if the group verifies that it's trustworthy, then it will cooperate".
"⊢□(□x→x)→x" says "it's verified that the group will cooperate on the basis of verified trustworthiness"

Now let's work through the proof in words, too! I'll omit saying "it's verified that..." each time, which is what ⊢ means.

⊢x→(□x→x), by tautology (A→(B→A)). This says:
"If the group cooperates, then it's trustworthy" (in the specific sense of trustworthiness about cooperation defined above).

⊢□x→□(□x→x), from 1 by □ necessitation and distributivity. This says:
"If the group verifiably cooperates, it's verifiably trustworthy."

⊢□(□x→x)→x, by assumption. This says:
"Assume the group will cooperate on the basis of verified trustworthiness."

⊢□x→x, from 2 and 3 by modus ponens. This says:
"The group is trustworthy."

⊢□(□x→x), from 4 by □ necessitation. This says:
"The group is verifiably trustworthy."

⊢x, from 5 and 3 by modus ponens. This says:
"The group cooperates."

Continuing to use "trustworthiness" in the sense above, the whole proof may be summarized as follows:

"If a group verifiably cooperates, it's verifiably trustworthy (to itself). Assume the group cooperates on the basis of verified trustworthiness. Then, it also cooperates on the basi...