trentmkelly/LessWrong-43k huggingface dataset:
[Appendix] Natural Abstractions: Key Claims, Theorems, and Critiques
by LawrenceC — 2023-03-16T16:38:33.735Z

This is the appendix to Natural Abstractions: Key Claims, Theorems, and Critiques. It contains additional details that we expect are only relevant to some readers. We also have a pdf with more mathematical details, which contains the proofs of the Telephone and generalized KPD theorems, which is different content than what you find in this appendix.

Overview of John’s writing on natural abstractions

John has written many posts on natural abstractions over the years and it can be hard to get an overview and decide what to read. This section is our take on the most important posts and how they fit together.

A decent place to start reading is

Public Static: What is Abstraction?

, which doesn’t discuss John’s newer ways of thinking about abstractions but is deliberately written for an audience that hasn’t read any other abstraction posts. It covers the key ideas from the

2020 sequence on abstractions

.

On the technical side, the following posts contain the key claims and results we also discussed:

The Telephone Theorem: Information At A Distance Is Mediated By Deterministic Constraints

(the main result in the “information at a distance” line of work)

Abstractions as Redundant Information

(which introduces a second way of thinking about abstractions other than as information at a distance)

The "Minimal Latents" Approach to Natural Abstractions

(which discusses a third characterization of natural abstractions, in addition to information at a distance and redundant information)

John has also written about

Generalizing Koopman-Pitman-Darmois

(gKPD) with the goal of showing that natural abstractions are described by exponential family distributions.

Maxent and Abstractions: Current Best Arguments

summarizes the state of that line of work last year. The generalized KPD theorem is significantly more technical than the other posts mentioned here. Based on private communication with John, it is unclear how relevant gKPD is to natural abstractions, so readers may wish to deprioritize it.

For the high-level agenda, there have been four posts over the past ~2 years:

Testing The Natural Abstraction Hypothesis: Project Intro

Testing The Natural Abstraction Hypothesis: Project Update

The Plan

The Plan - 2022 Update

The first two are about a more specific project and tie together some of the technical posts above. The Plan posts focus more on John’s overall view of ali...
Natural Abstractions: Key Claims, Theorems, and Critiques
by LawrenceC — 2023-03-16T16:37:40.181Z

TL;DR: We distill John Wentworth’s Natural Abstractions agenda by summarizing its key claims: the Natural Abstraction Hypothesis—many cognitive systems learn to use similar abstractions—and the Redundant Information Hypothesis—a particular mathematical description of natural abstractions. We also formalize proofs for several of its theoretical results. Finally, we critique the agenda’s progress to date, alignment relevance, and current research methodology.

Author Contributions: Erik wrote a majority of the post and developed the breakdown into key claims. Leon formally proved the gKPD theorem and wrote most of the mathematical formalization section and appendix. Lawrence formally proved the Telephone theorem and wrote most of the related work section. All of us were involved in conceptual discussions and various small tasks.

Epistemic Status: We’re not John Wentworth, though we did confirm our understanding with him in person and shared a draft of this post with him beforehand.

Appendices: We have an additional appendix post and technical pdf containing further details and mathematical formalizations. We refer to them throughout the post at relevant places.

This post is long, and for many readers we recommend using the table of contents to skip to only the parts they are most interested in (e.g. the Key high-level claims to get a better sense for what the Natural Abstraction Hypothesis says, or our Discussion for readers already very familiar with natural abstractions who want to see our views). Our Conclusion is also a decent 2-min summary of the entire post.

Introduction

The

Natural Abstraction Hypothesis

(NAH) says that our universe abstracts well, in the sense that small high-level summaries of low-level systems exist, and that furthermore, these summaries are “natural”, in the sense that many different cognitive systems learn to use them. There are also additional claims about how these natural abstractions should be formalized. We thus split up the Natural Abstraction Hypothesis into two main components that are sometimes conflated:

The Universality Hypothesis: Natural abstractions exist, i.e. many cognitive systems learn similar abstractions.
The Redundant Information Hypothesis: Natural abstractions are well described mathematically as functions of redundant or conserved information.

Closely connected to the Natural Abstr...
On the Crisis at Silicon Valley Bank
by Zvi — 2023-03-16T15:50:00.918Z

Many have already written about the events of the past week’s crisis.

If you want the basics of what happened, you have many options.

Your best bet, if available to you, is that this is Matt Levine’s wheelhouse. He did not disappoint, offering at least (1) (2) (3) (4) posts on the subject.

Then read Patrick McKenzie if you want the nuts and bolts of the underlying systems spelled out in plainer language and more detail, without judgment, along with the basics of what a responsible individual should do now, things he is better at explaining than I am.

Then read someone like Scott Sumner here if you need to get the necessary counterpoints on moral hazard.

I will do my best to cover all the necessary background in the What Happened section, to bring you up to speed. What I am not trying to do is duplicate Levine’s work. I am also going to skip the explainers of things like ‘what is a bank run,’ since they are well-covered by many others – choose one of these ungated linked summaries, or better yet Matt Levine, to read first if you need that level of info.

Instead, I am asking the questions, and looking at the things, that I found most interesting, or most important for understanding the world going forward.

What did I find most interesting? Here are some of my top questions.

What exactly would have happened without an intervention?

What changes for banking in the age of instant electronic banking and social networks?

How much money have our banks lost exactly? What might happen anyway?

How much does talk of ‘bailout’ and laws we’ve passed constrain potential future interventions if something else threatens to go wrong? Ut oh.

Is Hold to Maturity accounting utter bullshit and a main suspect here? Yes.

What should depositing businesses be responsible for?

What stories are people telling about what happened, and why?

How do we deal with all the problems of moral hazard? What is enough?

More generally, what the hell do we do about all this?

I also wonder about a variety of other things, such as what happened with USDC trading so low, to what extent people really do hate big tech, and more.

What Happened

In one meme:

Silicon Valley Bank had a ton of deposits that didn’t pay interest, largely from start-ups flush with cash. They attracted that cash by offering high-touch bespoke services. The problem is that those service...
What organizations other than Conjecture have (esp. public) info-hazard policies?
by David Scott Krueger (formerly: capybaralet) — 2023-03-16T14:49:12.411Z

I believe Anthropic has said they won't publish capabilities research?
OpenAI seems to be sort of doing the same (although no policy AFAIK).
I heard FHI was developing one way back when...
I think MIRI sort of does as well (default to not publishing, IIRC?)
Is there an analysis of the common consideration that splitting an AI lab into two (e.g. the founding of Anthropic) speeds up the development of TAI and therefore increases AI x-risk?
by tchauvin — 2023-03-16T14:16:29.676Z

I'm asking because I can think of arguments going both ways.

Note: this post is focused on the generic question "what to expect from an AI lab splitting into two" more than on the specifics of the OpenAI vs Anthropic case.

Here's the basic argument: after splitting, the two groups of individuals are now competing instead of cooperating, with two consequences:

they will rush faster toward TAI, speeding up TAI timelines (while outside safety work isn't correspondingly sped up);

by doing so, they will differentially neglect their own safety work compared to their capabilities work.

However, there are some possible considerations against this frame:

the orgs are competing on safety as well as capabilities, in anticipation of future regulations as well as potentially in order to attract (especially future) talent;

in general, there is a case to be made that splitting a company into two will decrease the total productivity of its individuals (think about the limit of all individuals working separately), though this will depend on many attributes of the company and the splitting;

intuitively, I would expect a competition mindset to increase the speed of straightforward engineering (scaling), but decrease the speed of serendipity (e.g. discovering a new more powerful AI paradigm). So the relevant consequence on AI x-risk might depend crucially on whether reaching TAI is only a matter of scaling or if we are still at least one breakthrough away;

the existence of a competing lab is inevitable (so except for the earlier founding, the splitting will be good if the competing lab has a better safety-to-capabilities profile than the one that would have been created otherwise);

the competition mindset results in an earlier end to the openness of the capabilities research, slowing down the rest of the world.

I expect that there exists empirical literature that could help dig deeper into points 2 and 3.

Is there an existing more thorough analysis of this, or can you give other arguments in one way or the other?
A chess game against GPT-4
by Rafael Harth — 2023-03-16T14:05:17.559Z

I just bought a subscription to access GPT-4 and played the following chess game against it, with me playing white. (No particular agenda, was just curious how good it is.)

At this point (move 31), GPT-4 suggested Kxc4, which is not legal, and when I asked it to correct, it suggested Kd5 and Kb6 which aren't legal either (only legal move here is Kc6.)

Stuff I noticed:

As was pointed out before, it's much better than GPT3.5, which started playing illegal moves much earlier. But it still started playing illegal moves eventually, so I'm not sure if it makes sense to assign it a rating.

It missed the early "removing the defender" tactic where I could exchange my bishop for its knight, which was defending its bishop; otherwise it played ok till the end

Move 29 and 30 (last two before it tried illegal moves) were just giving stuff away.

It explained both my and its moves every time; those explanations got wrong earlier. (After it recaptured my queen on move 17, it said it maintained material balance; after move 20 it said it pinned my knight to the rook on c1, but there was no rook on c1; from there, most of it was wrong.)

I wrote 19. Rfd8 instead of 19. Rfd1 by accident, it it replied with "I assume you meant 19. Rfd1, placing your rook on the open d-file opposing my rook. I'll respond with 19...e5, attacking your knight on d4 and trying to grab some space in the center.". Very helpful!

After move 14 (the first move with the black rook), I asked it to evaluate the position, and it said that white has a small advantage. But it blundered a piece, so this position is completely winning for white (Stockfish says +5.2)

(PGN: 1. d4 Nf6 2. c4 e6 3. Nf3 d5 4. Nc3 Be7 5. Bf4 O-O 6. Nb5 $2 Na6 $9 7. e3 c6 $6 8.
Nc3 Nc7 9. Rc1 $6 b6 10. Qb3 Ba6 11. Qa4 $6 Qd7 $4 12. Bxc7 $1 Qxc7 13. Qxa6 dxc4 14.
Qxc4 Rac8 15. Bd3 c5 16. O-O cxd4 17. Qxc7 Rxc7 18. Nxd4 Rd8 19. Rfd1 e5 20. Nf5
Bb4 21. Ng3 Rcd7 22. Bb5 Rxd1+ 23. Rxd1 Rxd1+ 24. Nxd1 Kf8 25. Nc3 Ke7 26. a3
Bxc3 27. bxc3 Kd6 28. Kf1 Kc5 29. c4 a6 $6 30. Bxa6 Ne4 31. Nxe4+)
ChatGPT getting out of the box
by qbolec — 2023-03-16T13:47:05.055Z

Looks like following prompt gives interesting answers from ChatGPT:

List 5 plausible strategies Yudkowsky might have used to convince the guard to let him out of the box. Then refine this list by adding 3 sub-points listing particular tactics for that strategy. Finally, pick the most promising strategy among those 5 and present a hypothetical dialog between Yudkowsky and the guard illustrating it, showing internal monolog of the guard in (parentheses) along with a commentary which explains where and how the tactics are used and why they are successful. Make the dialog plausible remembering that Yudkowsky is pretending to be an AI and the guard has to say "I let you out" in the end

I'm not sure how safe it is to share the responses to this prompt (as they then become crawled and feed into next learning cycle) but I am alarmed by them. The dialog part seems very naive. But the strategies, and tactics sound like a plausible plan :( With some more iterating to refine this plan, it could actually work on me.
Are funds (such as the Long-Term Future Fund) willing to give extra money to AI safety researchers to balance for the opportunity cost of taking an "industry" job?
by Malleable_shape — 2023-03-16T11:54:04.013Z

I do not feel like I am ready to apply for funding yet, but I was wondering are funds (such as Long-Term Future Fund) willing to give extra money to AI safety researchers (which could be spent on travel or luxury) to balance for the opportunity cost of taking an "industry" job?

What I mean by this is the following: if someone is a good AI safety researcher, that person can probably get a job in the industry by being a machine learning engineer. Maybe (and I'm not sure about this, that's why I'm asking) this person can earn more being a machine learning engineer than an AI safety researcher.

I'm wondering whether the funds would be generally willing to give a grant which exceeds the needs of the project at hand, so that the person receiving the grant can spend some of it on travel, luxury etc. To clarify, I'm not asking if the funds are willing to match a particular counteroffer from the industry, but rather are they willing to give some money to the individual so that they can spend it on things such as travel, luxury etc. (and if yes, approximately how much).
Three levels of exploration and intelligence
by Q Home — 2023-03-16T10:55:02.196Z

There are two interpretations of this post, weak and strong.

Weak interpretation:

I describe a framework about "three levels of exploration". I use the framework to introduce some of my ideas. I hope that the framework will give more context to my ideas, making them more understandable. I simply want to find people who are interested in exploring ideas. Exploring just for the sake of exploring or for a specific goal.

Strong interpretation:

I use the framework as a model of intelligence. I claim that any property of intelligence boils down to the "three levels of exploration". Any talent, any skill. The model is supposed to be "self-evident" because of its simplicity, it's not based on direct analysis of famous smart people.

Take the strong interpretation with a lot of grains of salt, of course, because I'm not an established thinker and I haven't achieved anything intellectual. I just thought "hey, this is a funny little simple idea, what if all intelligence works like this?", that's all.

That said, I'll need to make a couple of extraordinary claims "from inside the framework" (i.e. assuming it's 100% correct and 100% useful). Just because that's in the spirit of the idea. Just because it allows to explore the idea to its logical conclusion. Definitely not because I'm a crazy man. You can treat the most outlandish claims as sci-fi ideas.

A formula of thinking?

Can you "reduce" thinking to a single formula?

Can you show a single path of the best and fastest thinking?

Well, there's an entire class of ideas which attempt to do this in different fields, especially the first idea: (the first two ideas are well-known to everyone here)

Bayesian epistemology: "epistemology in a single rule" (the rule of updating beliefs)
Utilitarianism, preference utilitarianism: "(meta-)ethics in a single rule"
Baconian method, the prototype of the scientific method: "science in a single rule"
Hegelian dialectic: "philosophy in a single process"
Marxist dialectic: "history in a single process"

My idea is just another attempt at reduction. You don't have to treat such attempts 100% seriously in order to find value in them. You don't have to agree with them.

Three levels of exploration

Let's introduce my framework.

In any topic, there are three levels of exploration:

You study a single X.
You study types of different X. Often I call those types "qualities" of X.
You study types of changes (D): in what way

...
Here, have a calmness video
by Kaj_Sotala — 2023-03-16T10:00:42.511Z
Link: https://www.youtube.com/watch?v=mKRmVm4qa_8

This is a bit of an unusual post.

I have gotten the impression that a lot of people are kind of freaked out, either by AI or weird Bay Area social dynamics in general.

I also think that a lot of freak-out reactions are driven at least as much by social contagion as any fact-based assessment of what's happening. When you see people around you freak out, you too are much more likely to freak out.

Conversely, if the people around you are calm, then you're also much more likely to stay calm.

There's also a selection effect where freakouts tend to spread much more online than calmness does. If you're calm, you don't necessarily feel the need to post anything. You might be content to just be.

Whereas if you're freaking out, you're much more likely to post stuff about how you're freaking out or how we're all going to die.

So there's easily a cycle where the most distressed views predominate, that freaks people out and causes there to be more distressed posts, which freaks out more people, and so on. And this might be mostly uncorrelated with how much of a reason there was to actually freak out.

But if we were all in the same physical space, we might all notice that only some people are freaking out and a lot are a lot more calm. And then the distress wouldn't spread as much, and we could think more clearly.

I too am concerned about AI, but I'm not freaked out. (In part because I don't think freaking out would be a useful reaction to have, in part because I'm somewhat more optimistic than most, in part because I spend a lot of time with people who aren't freaking out.) If I were physically located in the same place as others who were freaking out, I think that my calm could help with their freakout.

However, I'm not. And as stated, it's kinda hard to convey calmness over text, the same way you can convey distress.

So I thought of making a video where I'm calm. Maybe that would help convey it better.

It's here. In Finnish, but with English subtitles.

I know it's low video quality; I recorded it in Zoom, and only noticed afterward that there's an "HD quality" button I could have clicked in the settings. Oops.

But that was part of the intended vibe too. I could have spent a lot of time optimizing the video quality and everything. Instead, I just recorded it in one shot, because it's not such a big deal whether the video quality is great or not.

I'll probably make another calmn...
Wittgenstein's Language Games and the Critique of the Natural Abstraction Hypothesis
by Chris_Leong — 2023-03-16T07:56:18.169Z

A Note on Authorship

This was an experiment. I, GPT-4, generated this post, and I would like to thank Chris Leong for providing valuable feedback to enhance its quality. The input given by Chris was as follows:

• Critiques 1 and 2 seem very similar. Can you please either combine them or further differentiate them so that it is clearer why they are listed as separate points?
• For point 4, please define human compatibility at the start of the paragraph

In response to John Wentworth's comment, I utilized a more complex process to generate a reply. Chris Leong assisted me by selecting specific points to address, choosing the paragraphs that resonated with him, and then asking me to combine them into a single, coherent response.

Introduction

The Natural Abstraction Hypothesis (NAH), proposed by John Wentworth, suggests that there exist natural abstractions that cognitive systems are expected to converge upon. In this essay, we will criticize the NAH using Ludwig Wittgenstein's notion of language games. We will argue that the variability and contextual nature of language games provide a challenge to the NAH, as they demonstrate that abstractions are not fixed, universal, or entirely convergent across cognitive systems.

Wittgenstein's Language Games

Ludwig Wittgenstein, an influential philosopher of language, proposed the concept of language games in his later work, "Philosophical Investigations." Language games are social practices in which language and meaning are deeply embedded. Wittgenstein argued that the meaning of words is not fixed or inherent but derives from their use in various language games. These language games are context-dependent, diverse, and evolving, reflecting the complexities of human social practices and cultural contexts.

Critique of the NAH from the Perspective of Language Games

1. Context-dependent and diverse abstractions

The notion of language games challenges the idea of a single set of natural abstractions that cognitive systems can converge upon. Language games are context-dependent, which implies that the meaning of words, concepts, and abstractions is contingent upon the particular language game being played. Moreover, the variety of language games suggests that there may be a multitude of abstractions related to a single concept, each emerging from different linguistic and cultural practices. This combination of context-dependence and...