trentmkelly/LessWrong-43k huggingface dataset:
Conservation of Expected Evidence and Random Sampling in Anthropics
by Ape in the coat — 2023-09-03T06:55:10.003Z

This is the second post in my series on Anthropics. The previous one is Anthropical Motte and Bailey in two versions of Sleeping Beauty. The next one is Antropical probabilities are fully explained by difference in possible outcomes.

Introduction

Since the first time I've heard about anthropics something felt off. Be it updating on awakening in Sleeping Beauty, accepting high probability of doom in the Doomsday Argument, or the premise of Grabby Aliens; following SSA or SIA, the whole pattern of reasoning seemed wrong to me.

That it's cheating. That there is something obviously unlawful going on. That it doesn't look at all like the way cognition engines produce map-territory correspondence.

It was hard to point out what exactly had been wrong, though. The discourse seemed to be focused around either accepting SIA or SSA, both of which are obviously absurd and wrong in some cases. Or discarding all the anthropic reasoning altogether - a position towards which I'm quite sympathetic, but which also seemed to be an overcorrection, as it tends to discard some sound reasoning.

It took me some time to formalize this feeling of wrongness into an actual rule that can be applied to anthropical problems, separating correct reasoning from wrong reasoning. In this post I want to explore this principle and its edge cases, and show how using it can make anthropics add up to normality.

The metaphysics of existence vs having a blue jacket

Let's consider the Blue Jacket Experiment (BJE):

A fair coin is tossed. On Tails two people will be created, wearing blue jackets. On Heads two people will also be created but only one, randomly chosen, will have a blue jacket. You are created and notice that you are wearing a blue jacket. What's the probability that the coin landed Heads?

Here the situation is quite obvious. As having a blue jacket is twice as likely when the coin is Tails, we can do a simple Bayesian update:

P(Heads|BlueJacket)=P(BlueJacket|Heads)P(Heads)/P(BlueJacket)=1/3

Notice that this is not an Anthropical Motte where we count Tails outcomes twice for the same coin toss. In a repeated experiment guessing Tails when you have a blue jacket gives about 2/3 accuracy. You can actually guess Tails per experiment better than chance this way.

So why doesn't the same principle apply to the Incubator Sleeping Beauty (ISB) problem, as one would naively think? Why can't I ...
The goal of physics
by Jim Pivarski — 2023-09-02T23:08:02.125Z

In grad school, I was a teaching assistant for a course called, Why the Sky is Blue. It was a qualitative introduction to physics for non-majors, covering a lot of the same topics as Physics I, such as forces, conservation of energy and momentum, electric charges and magnetic fields, in less detail, with not much math. The actual question about why the sky is blue was saved for the end. As the course dragged on and the students (who expected no math, rather than not much math) started to complain, "Are we ever going to find out why the sky is blue?" I watched the schedule slip and wondered the same thing.

We skipped some sections and managed to wedge it into the last lecture: finally, we were talking about why the sky is blue! "The sky is blue because of Rayleigh scattering." Okay, that's not an answer we hadn't defined Rayleigh scattering, there wasn't time for it, so we said that air molecules absorb and re-radiate—effectively changing the direction of—blue light more than red light. Red light goes straight through the atmosphere, and blue light bounces around, making the whole sky glow blue. Conversely, sunrises and sunsets are red because you're looking at the light that has gone straight through a larger wedge of atmosphere. It lost most of its blue on the way to your eye.

Pretty good explanation, for not being able to say

I=I01+cos2θ2R2(2πλ)4(n2−1n2+2)2(d2)6

(the 1/λ4 part affects small-λ blue light more than large-λ red light). We also showed pictures like this sunset:

to demonstrate the effect of straight-through red light and bouncing-around blue light.

So in the end, "Why is the sky blue?"

Answer: "Because sunsets are red!"

"And why are sunsets red...?"

It was understandably unsatisfying. One thing was only explained in terms of another thing. But even if we had the time to get into detail about Rayleigh scattering, they could reasonably ask, "Why does light scatter according to that formula?" We could go deeper and explain Lord Rayleigh's proof in terms of Maxwell's equations. And whyfore Maxwell's equations? Well, quantum electrodynamics, which is a quantum field theory with a local U(1) gauge symmetry, which is to say that every point in space has an extra degree of freedom, similar to a fourth spatial dimension except that this dimension can't be rotated with normal space like the other three, this dimension is connected to...
PIBBSS Summer Symposium 2023
by Nora_Ammann — 2023-09-02T17:22:44.576Z

Tl;dr

We are pleased to invite you to the PIBBSS Summer Symposium where the fellows from the ‘23 fellowship program present their work.

The symposium is taking place online, over several days in the week of September 18th.

Check out the

full program

,

including brief descriptions for each talk. (You can toggle between the different days.)

Register

via this link

. You will receive an email with the event link which you can use for the entire symposium. You will be able to drop into as few or as many presentations and breakout sessions as you like.
Click here to add the schedule to your google calendar.

About PIBBSS

PIBBSS

is a research initiative aiming to explore parallels between intelligent behavior in natural and artificial systems, and to leverage these insights towards the goal of building safe and aligned AI.

During June-August ‘23, we ran the second iteration of our 3-month

research fellowship program

. The symposium acts as a venue to share research conducted as part of this program.

About the symposium

The PIBBSS Summer Symposium is a multi-day event where PIBBSS fellows present their work.

The event is taking place on the days of Tuesday - Friday, Sept 19th - 22nd, between 18:00-~21:00 CET / 9:00 - 12:00 PT / 12:00 - 15:00 ET.

The event is set it up such that you can easily join whichever talks and breakout sessions you are most interesting in.

The program

Find a program overview here.

Find the

full program

here

, including brief descriptions of each talk.

On top of the talks, there will also be opportunities to continue the discussion with fellows at the end of each block in speaker-specific breakout rooms.

Talks span a wide range of topics in line with PIBBSS’s research mission. Some representative examples of topics include:

novel avenues for interpretability
naturalistic approaches to understanding the nature and emergence of agency and goal-directed behavior
attempts to develop a principled understanding of the dynamics emerging from multi-agent interactions in and across AI/LLM systems
analyses of the space of AI risks — from single to multiple agents, from misuse to structural risks, etc.
exploration of the potential and limits of existing legal tools for reducing catastrophic risks from AI
..and more!

The format

The symposium is taking place over the course of...
The smallest possible button (or: moth traps!)
by Neil  — 2023-09-02T15:24:20.453Z
Link: https://neilwarren.substack.com/p/the-smallest-possible-button

tl;dr: The more knowledge you have, the smaller the button you need to press to achieve desired results. This is what makes moth traps formidable killing machines, and it's a good analogy for other formidable killing machines I could mention.

Traps

I was shopping for moth traps earlier today, and it struck me how ruthlessly efficient humans could be in designing their killing apparatus. The weapon in question was a thin pack in my hands containing just a single strip of paper which, when coated with a particular substance and folded in the right way, would end up killing most of the moths in my house. No need to physically hunt them down or even pay remote attention to them myself; a couple bucks spent on this paper and a minute to set it up, and three quarters of the entire population is decimated in less than a day.

That’s… horrifying.

Moth traps are made from cardboard coated with glue and female moth pheromones. Adult males are attracted to the pheromones, and end up getting stuck to the sides where they end up dying.[1] The females live, but without the males, no new larvae are born and in a few months time you’ve wiped out a whole generation of moths.[2] These traps are “highly sensitive” meaning that they will comb a whole room of moths very quickly despite being passive in nature.

Why are moth traps so effective? They use surgically precise knowledge. Humans know how to synthesize moth pheromones, and from there you can hack a 250-million-year-old genetically derived instinct that male moths have developed for mating, and then you set a trap and voilà. The genetic heuristic that worked 99% of the time for boosting reproductive rates in moths can be wielded against moths by obliterating their reproductive rates.

Moth traps aren’t even the pinnacle of human insecticidal war machines. Scientists have, after all, seriously considered using gene drives to eliminate an entire species of mosquitoes with a single swarm and some CRISPy cleverness.[3]

The smallest button

Moth traps and gene drives work by understanding something so well that when you use brute force (because everything is brute force) to do something, you do it in the most optimal and surgical way. Intelligent design means humans can engineer very, very effective traps that harness the smallest buttons you can push in order to get a desired result.

Evolution can also produce sexua...
Steven Harnad: Symbol grounding and the structure of dictionaries
by Bill Benzon — 2023-09-02T12:28:05.125Z

Cross-posted from New Savanna.

Stevan Harnad: AI's Symbol Grounding Problem, The Gradient podcast, August 31, 2023

Stevan Harnad is professor of psychology and cognitive science at Université du Québec à Montréal, adjunct professor of cognitive science at McGill University, and professor emeritus of cognitive science at the University of Southampton. His research is on category learning, categorical perception, symbol grounding, the evolution of language, and animal and human sentience (otherwise known as “consciousness”). He is also an advocate for open access and an activist for animal rights.

Outline:

(00:00) Intro
(05:20) Professor Harnad’s background: interests in cognitive psychobiology, editing Behavioral and Brain Sciences

(07:40) John Searle submits the Chinese Room article
(09:20) Early reactions to Searle and Prof. Harnad’s role

(13:38) The core of Searle’s argument and the generator of the Symbol Grounding Problem, “strong AI”
(19:00) Ways to ground symbols
(20:26) The acquisition of categories
(25:00) Pantomiming, non-linguistic category formation
(27:45) Mathematics, abstraction, and grounding
(36:20) Symbol manipulation and interpretation language
(40:40) On the Whorf Hypothesis
(48:39) Defining “grounding” and introducing the “T3” Turing Test
(53:22) Turing’s concerns, AI and reverse-engineering cognition
(59:25) Other Minds, T4 and zombies
(1:05:48) Degrees of freedom in solutions to the Turing Test, the easy and hard problems of cognition
(1:14:33) Over-interepretation of AI systems’ behavior, sentience concerns, T3 and evidence sentience
(1:24:35) Prof. Harnad’s commentary on claims in The Vector Grounding Problem
(1:28:05) RLHF and grounding, LLMs’ (ungrounded) capabilities, syntactic structure and propositions
(1:35:30) Multimodal AI systems (image-text and robotic) and grounding, compositionality
(1:42:50) Chomsky’s Universal Grammar, LLMs and T2
(1:50:55) T3 and cognitive simulation
(1:57:34) Outro

The podcast site also has links to Harnad’s webpages and to five selected articles. One of them in particular, about the structure of dictionaries, interested me. Here’s the citation, abstract, and a link:

Philippe Vincent-Lamarre, Alexandre Blondin Massé, Marcos Lopes,Mélanie Lord, Odile Marcotte, Stevan Harnad. The Latent Structure of Dictionaries. Topics in Cognitive Science 8 (2016) 625–659. DOI: 10.1111/tops.12211. (Open Access)

Abstract: How many words—and which

...
Is Metaethics Unnecessary Given Intent-Aligned AI?
by Caleb Biddulph — 2023-09-02T09:48:54.934Z

While you can read this post as a standalone, I'm writing it partially in response to Wei Dai's recent post Meta Questions about Metaphilosophy. I had already been considering writing a top-level post about metaethics and whether "solving" it is a tractable problem that can meaningfully be separated from AI alignment itself, and I see Wei's post as closely related to this topic. I recommend reading that (quite short) post too!

I define metaethics as the problem of how to decide on a "correct" ethical system. As an example, it seems like a good idea to give every "person" a "vote" about what ethical system they "want." But what if people say they want different things depending on how they're asked? How do we define a person? What voting system do we use? Metaethics is about finding answers to questions like these. Coherent Extrapolated Volition (CEV) is one attempt at a solution to metaethics. So the argument goes, solving AI alignment lets us put an arbitrary goal into an AI, but solving metaethics tells us what the goal should be.

I believe that metaethics is at least a major part of what Wei referred to as "metaphilosophy" in his post, although he also seems to be concerned about other things, like how to decide which decision theory to use. In this post, I'll only be talking about metaethics, as I'm not sure I understand Wei's concerns about other aspects of metaphilosophy enough to comment on them.

I used to feel more strongly about it being highly important to "solve metaethics" before we get transformative AI. Then I talked to someone who argued against this by basically saying "if we have an aligned AI that's at least as good as you at metaethics, we can just tell it to do all the thinking you're doing now/." After thinking about it some more, I mostly convinced myself that they were right. In this post, I'll give a more refined version of the argument for why we don't need to "solve metaethics," then explain the strongest remaining argument against the "just solve intent alignment" strategy that I know of.

Why solving metaethics might be unnecessary

Here's a concern I believe Wei Dai and others would endorse:

"Even if we solve intent alignment, our superintelligent AI may end up settling on bad ethics."

(With the proposed solution being to solve metaethics before we build the AI.)

Let's try to break this statement down, starting with what we mean by "...
Rational Agents Cooperate in the Prisoner's Dilemma
by Isaac King — 2023-09-02T06:15:15.720Z

This is a crosspost from my blog. Its intention is to act as a primer on non-causal decision theories, and how they apply to the prisoner's dilemma in particular, for those who were taught traditional game theory and are unfamiliar with the LessWrong memeplex. I'm posting it here primarily to get feedback on it and make sure I didn't make any errors. I plan to update it as I hear counterarguments from causal decision theorists, so check the one on my blog if you want the most recent version.

The Prisoner's Dilemma is a well-known problem in game theory, economics, and decision theory. The simple version: Alice and Bob have been arrested on suspicion of having committed a crime. If they both stay silent, they'll each do a year in prison. If one of them testifies against the other while the other stays silent, the one silent one goes to prison for 4 years and the narc goes free. If they both testify against each other, they each serve two years in prison. In order to simplify the math, game theorists also make three rather unrealistic assumptions: the players are completely rational, they're completely self-interested, and they know all of these details about the game.

There's a broad consensus in these academic fields that the rational choice is to defect. This consensus is wrong. The correct choice in the specified game is to cooperate. The proof of this is quite straightforwards:

Both agents are rational, meaning that they'll take the action that gets them the most of what they want. (In this case, the fewest years in jail.) Since the game is completely symmetric, this means they'll both end up making the same decision. Both players have full knowledge of the game and each other, meaning they know that the other player will make the rational choice, and that the other player knows the same thing about them, etc. Since they're guaranteed to make the same decision, the only possible outcomes of the game are Cooperate-Cooperate or Defect-Defect, and the players are aware of this fact. Both players prefer Cooperate-Cooperate to Defect-Defect, so that's the choice they'll make.

Claiming that an entire academic field is wrong is a pretty bold claim, so let me try to justify it.

How an entire academic field is built on an incorrect assumption

The standard argument for Defecting is that no matter what the other player does, you're better off having defected. ...
[Linkpost] Large language models converge toward human-like concept organization
by Bogdan Ionut Cirstea — 2023-09-02T06:00:45.504Z

This is a linkpost for https://arxiv.org/abs/2308.15047.

Large language models show human-like performance in knowledge extraction, reasoning and dialogue, but it remains controversial whether this performance is best explained by memorization and pattern matching, or whether it reflects human-like inferential semantics and world knowledge. Knowledge bases such as WikiData provide large-scale, high-quality representations of inferential semantics and world knowledge. We show that large language models learn to organize concepts in ways that are strikingly similar to how concepts are organized in such knowledge bases. Knowledge bases model collective, institutional knowledge, and large language models seem to induce such knowledge from raw text. We show that bigger and better models exhibit more human-like concept organization, across four families of language models and three knowledge graph embeddings.

We present a series of experiments with four families of LLMs (21 models), as well as three knowledge graph embedding algorithms. Using three different methods, we compare the vector spaces of the LLMs to the vector spaces induced by the graph embedding algorithms. (This amounts to a total of 220 experiments.) We find that the vector spaces of LLMs within each family become increasingly structurally similar to those of knowledge graph embeddings. This shows that LLMs partially converge on human-like concept organization, facilitating inferential semantics [19].3 The sample efficiency of this convergence seem to depend on a number of factors, including polysemy and semantic category. Our findings have important implications. They vindicate the conjecture in [19] that LLMs exhibit inferential semantics, settling the research question presented in [16], cited above. This means that LLMs partially converge toward human-like concept representations and, thus, come to partially ‘understand language’, in a non-trivial sense. We speculate that the human-like conceptual organization is also what facilitates out-of-distribution inferences in LLMs.
Plum Cooking Temperature
by jefftk — 2023-09-02T01:30:03.086Z

After having fun with cooking/drying plums

last year, I decided to have another go
this year. This time I compared two different ways of cooking whole
plums: 140F for ~1.5d vs 350F for ~1.5hr.

I was curious whether the higher temperature was better or worse.
Maybe interesting flavors are lost with higher temperatures, or maybe
they're lost with longer time? Maybe these approaches differ in how
they convert starch to sugar?

Here's a picture of the 350F plums as I was removing them from their
tray:

After taking them out I manually removed the pits, and I made sure to
scrape the pan with a spatula to get the tasty juice.

Here are ready-for-rebagging
pictures of both, 350F on the left and 140F on the right:

I did some testing, where people didn't know which was which:

Overall everyone liked the 350F better: richer plum flavor. One
taster thought the 140F had a bit of an "off" flavor. The 350F was
less sweet, enough so that for many dishes you wouldn't want to use it
straight, but adding sugar to taste later isn't hard.

(The 350F is also way easier logistically—tying up the oven for
over a day isn't great housemate behavior.)
What did you learn from leaked documents?
by wassname — 2023-09-02T01:28:47.120Z

As rationalists we seek to understand the world, but it's made harder by political bias and different agendas.

Leaked documents, represent a kind of ground truth, showing how the world really works. Telling us what's for sale, what the real agendas are, how powerful spies are, and how coordinated governments are. They are almost the opposite to conspiracy theories, as they present observations that can prune conspiracy theories.

But there are too many documents to read, so let's compare notes. What surprised you and caused you to update your view of the world?

Update: related conversation on TheMotte.org
One Minute Every Moment
by abramdemski — 2023-09-01T20:23:56.391Z

About how much information are we keeping in working memory at a given moment?

"Miller's Law" dictates that the number of things humans can hold in working memory is "the magical number 7±2". This idea is derived from Miller's experiments, which tested both random-access memory (where participants must remember call-response pairs, and give the correct response when prompted with a call) and sequential memory (where participants must memorize and recall a list in order). In both cases, 7 is a good rule of thumb for the number of items people can recall reliably.[1]

Miller noticed that the number of "things" people could recall didn't seem to depend much on the sorts of things people were being asked to recall. A random numeral contains about 3.3 bits of information, while a random letter contains about 4.7; yet people were able to recall about the same number of numerals or letters.

Miller concluded that working memory should not be measured in bits, but rather in "chunks"; this is a word for whatever psychologically counts as a "thing".

This idea was further reinforced by memory athletes, who gain the ability to memorize much longer strings of numbers through practice. A commonly-repeated explanation is as follows: memory athletes are not increasing the size of their working memory; rather, they are increasing the size of their "chunks" when it comes to recalling strings of numbers specifically.[2] For someone who rarely needs to recall numbers, individual numerals might be "chunks". For someone who recalls numbers often due to work or hobby, two or three-digit numbers might be "chunks". For a memory athlete who can keep hundreds of digits in mind, perhaps sequences of one hundred digits count as a "chunk".[3]

However, if you're like me, you probably aren't quite comfortable with Miller's rejection of bits as the information currency of the brain. The brain isn't magic. At some level, information is being processed.

I'll run with the idea that chunking is like Huffman codes. Data is compressed by learning a dictionary mapping from a set of "codewords" (which efficiently represent the data) to the decompressed representation. For example, if the word "the" occurs very frequently in our data, we might assign it a very short codeword like "01", while rare words like "lit" might get much longer codewords such as "1011010".

A codeword is sort of like a...