trentmkelly/LessWrong-43k huggingface dataset:
AI #85: AI Wins the Nobel Prize
by Zvi — 2024-10-10T13:40:07.286Z

Both Geoffrey Hinton and Demis Hassabis were given the Nobel Prize this week, in Physics and Chemistry respectively. Congratulations to both of them along with all the other winners. AI will be central to more and more of scientific progress over time. This felt early, but not as early as you would think.

The two big capability announcements this week were OpenAI’s canvas, their answer to Anthropic’s artifacts to allow you to work on documents or code outside of the chat window in a way that seems very useful, and Meta announcing a new video generation model with various cool features, that they’re wisely not releasing just yet.

I also have two related corrections from last week, and an apology: Joshua Achiam is OpenAI’s new head of Mission Alignment, not of Alignment as I incorrectly said. The new head of Alignment Research is Mia Glaese. That mistake it mine, I misread and misinterpreted Altman’s announcement. I also misinterpreted Joshua’s statements regarding AI existential risk, failing to take into account the broader context, and did a poor job attempting to reach him for comment. The above link goes to a new post offering an extensive analysis of his public statements, that makes clear that he takes AI existential risk seriously, although he has a very different model of it than I do. I should have done better on both counts, and I am sorry.

Table of Contents

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. Proofs of higher productivity.

Language Models Don’t Offer Mundane Utility. Why the same lame lists?

Blank Canvas. A place to edit your writing, and also your code.

Meta Video. The ten second clips are getting more features.

Deepfaketown and Botpocalypse Soon. Assume a data breach.

They Took Our Jobs. Stores without checkouts, or products. Online total victory.

Get Involved. Princeton, IAPS, xAI, Google DeepMind.

Introducing. Anthropic gets its version of 50% off message batching.

AI Wins the Nobel Prize. Congratulations, everyone.

In Other AI News. Ben Horowitz hedges his (political) bets.

Quiet Speculations. I continue to believe tradeoffs are a (important) thing.

The Mask Comes Off. What the heck is going on at OpenAI? Good question.

The Quest for Sane Regulations. The coming age of the AI agent.

The Week in Audio. Elon Musk is (going on) All-In until he wins, or l

...
Behavioral red-teaming is unlikely to produce clear, strong evidence that models aren't scheming
by Buck — 2024-10-10T13:36:53.810Z

One strategy for mitigating risk from schemers (that is, egregiously misaligned models that intentionally try to subvert your safety measures) is behavioral red-teaming (BRT). The basic version of this strategy is something like: Before you deploy your model, but after you train it, you search really hard for inputs on which the model takes actions that are very bad. Then you look at the scariest model actions you found, and if these contain examples that are strong evidence that the model is scheming (or has other behavioral problems), you don't deploy it.

BRT is likely to be a useful component of a good overall plan for handling risk from scheming. But though I think it’s plausible that BRT is able to give us strong evidence that our models are scheming, I don’t think it’s going to be able to give us very strong evidence that they aren’t.

A big theme in this piece is fear, uncertainty, and doubt, commonly abbreviated FUD. I think that BRT as a safety case is plagued by FUD—there are lots of different places where the approach relies on assumptions about the properties of scheming models that, while plausible, we won’t be confident enough to rely on.

Throughout this document, I’m writing under a somewhat nearcasted assumption—that is, I assume that the techniques we’re discussing get a few years of progress and so get incrementally better at a comparable rate to the rate of progress over the last few years, but I don’t assume that there are any radical breakthroughs. I think this is a perspective that is healthy for a few reasons. To start with, one of the main reasons to spend time now arguing about safety techniques to use for powerful AI is because it’s plausible that we’ll get very powerful AI soon, so it’s potentially very important to have solid plans for that eventuality. I talk to a lot of people at AI labs, and some have safety plans that rely on currently nonexistent techniques. For these plans, I want an argument that we’re on track to develop the prerequisite techniques, and I believe it would be prudent to also have plans that don’t rely on nonexistent prerequisites. But even aside from the possibility of short AI timelines, I think that the nearcasted approach lets me do more specific analysis of safety techniques, because it’s easier to discuss the potential safety properties of techniques we have now than hypothetical and usually...
Joshua Achiam Public Statement Analysis
by Zvi — 2024-10-10T12:50:06.285Z

Joshua Achiam is the OpenAI Head of Mission Alignment

I start off this post with an apology for two related mistakes from last week.

The first is the easy correction: I incorrectly thought he was the head of ‘alignment’ at OpenAI rather than his actual title ‘mission alignment.’

Both are important, and make one’s views important, but they’re very different.

The more serious error, which got quoted some elsewhere, was: In the section about OpenAI, I noted some past comments from Joshua Achiam, and interpreted them as him lecturing EAs that misalignment risk from AGI was not real.

While in isolation I believe this is a reasonable way to interpret this quote, this issue is important to get right especially if I’m going to say things like that. Looking at it only that way was wrong. I both used a poor method to contact Joshua for comment that failed to reach him when I had better options, and I failed to do searches for additional past comments that would provide additional context.

I should have done better on both counts, and I’m sorry.

Indeed, exactly because OpenAI is so important, and to counter the potential spread of inaccurate information, I’m offering this deep dive into Joshua Achiam’s public statements. He has looked at a draft of this to confirm it has no major errors.

Here is a thread Joshua wrote in November 2022 giving various links to AI safety papers and resources. The focus is on concrete practical ‘grounded’ stuff, also it also includes a course by Dan Hendrycks that involves both levels.

Joshua Achiam Has a Very Different Model of AI Existential Risk

Having looked at many additional statements, Joshua clearly believes that misalignment risk from AGI is real. He has said so, and he has been working on mitigating that risk. And he’s definitely been in the business many times of pointing out when those skeptical of existential risk get sufficiently far out of line and make absolute statements or unfair personal or cultural attacks.

He does appear to view some models and modes of AI existential risk, including Yudkowsky style models of AI existential risk, as sufficiently implausible or irrelevant as to be effectively ignorable. And he’s shown a strong hostility in the x-risk context to the rhetoric, arguments tactics and suggested actions of existential risk advocates more broadly.

So for example we have these:

Joshua Achiam (Mar

...
Do you want to do a debate on youtube? I'm looking for polite, truth-seeking participants.
by Nathan Young — 2024-10-10T09:32:59.162Z

I am testing a new debate format and am looking for people to debate any of the following (or make a case for other things):

Policy decisions
EA Cause prioritisation
What should/will happen in Taiwan/Ukraine/Gaza
US elections
Economics
Animal consciousness

The debate would happen via a video call and then be put on my youtube (and I guess other places if you want).

Why am I asking for volunteers?

Most debates are bad, at least in part because the people who have them just want to win. Go on youtube and you can find 10s of debates with bad faith arguments and a descent into insults. These debates are usually about either US politics, trans issues or religion.

I think a debate will go much better if it starts with a pair of people who both have some moderate care for the truth and have reputation locked into being open and polite. The work of having good discussions is hard enough, without making it harder.

What is the format?

I am testing a new format, which I ran at Manifest (and which many seemed to enjoy[1]):

Poll the two people debating
Find things they agree and disagree on
Introduce the debaters and the things they agree on
Take a specific area they disagree on
Each debater gets [2 x 5] minutes where they are in control. They can speak or ask questions
As the moderator, I uphold their boundaries during this time - they can't be interrupted.
Sometimes I will write key cruxes on a shared whiteboard
After the time is up for that topic, we move onto another specific disagreement
There is no mechanism to figure out who has "won"

Optiona additions I might try later.

@Katja Grace might sometimes help and try and disentangle cruxes. I am interested in having active moderation[2]
Give both candidates M10,000 on manifold and require that they bet it during the debate on markets relating to the debate which I will create
Some kind of live LLM based fact checking
Some kind of "throw to the audience" fact checking if I do a livestream

So do you have a view on these topics and want to discuss it?

The format aims to be relaxed and not require you to manage your own boundaries. If I do my job well, you don't have to worry about someone being rude, taking more time than allotted or shifting the question around. I hope this might allow participants more space to focus on the discussion.

^

Including @Eliezer Yudkowsky who asked if I could run one with him. He seems pretty hard to please.

^

I

...
Rationalist Gnosticism
by tailcalled — 2024-10-10T09:06:34.149Z

Gnosticism was an old Christian sect which solved the problem of evil by asserting that the material world was created by an evil entity, and that only inner enlightment can reveal the path to the true, good and loving god. I think many rationalists have a worldview similar to this, perhaps most strongly illustrated by The Goddess Of Everything Else, which portrays brutal competition dynamics as the creator of the world's forms, which just barely/unreliably yields "good values" due to the deeper realization that cooperation is possible.

I asked ChatGPT to make an image that could win an art competition, and it chose to generate this. This is the aesthetic I associate with Gnosticism: the sun is obscured and the darkness reveals beautiful lights giving a sense of some mysterious greater thing that will join forces with you if you follow it.

Things I think of as Gnostic:

Deriving morality from game theory, e.g. studying iterated prisoner's dilemma, tit-for-tat, etc..
Modelling latent variables without a material root cause analysis.
Studying what things "really are like" (often with reductionist methods), as distinct from their applications. Treating macroscopic objects as an abstract simplified representation of large numbers of microscopic objects, rather than vice versa.
Self-reference and especially attempts to solve paradoxes of self-reference using math/logic.
Paying more attention to Bayesian decision theory, economics or evolution/genetics than to social events, business or ecology. Starting with common/informative factors rather than powerful factors.
Trying to eliminate errors or eliminate biases, especially if one doesn't know where those errors/biases come from, or has derived the source of the errors/biases from theories like evolutionary psychology.

I sometimes place religions on a spectrum with varying degrees of emphasis on the material world vs personal inner enlightenment:

Paganism <- Judaism - Islam - Protestantism - Atheism - Gnosticism -> Buddhism

That said, this is quite stereotyped, and each religion has more-Gnostic variants (e.g. Judaism has Kabbalah) and less-Gnostic variants (e.g. Protestantism has the prosperity gospel). I see Paganism, especially sun worship, as the opposite of Gnosticism. Notably, The Goddess Of Everything Else does not even mention the sun. While a big reason I bring up Gnosticism and Paganism is to highlight the flaw...
Values Are Real Like Harry Potter
by johnswentworth — 2024-10-09T23:42:24.724Z

Imagine a TV showing a video of a bizarre, unfamiliar object - let’s call it a squirgle. The video was computer generated by a one-time piece of code, so there's no "real squirgle" somewhere else in the world which the video is showing. Nonetheless, there's still some substantive sense in which the squirgle on screen is "a thing" - even though I can only ever see it through the screen, I can still:

Notice that the same squirgle is shown at one time and another time.
Predict that a shiny spot on the squirgle will still be there later.
Discover new things about the squirgle, like a scratch on it.

… and so forth. The squirgle is still “a thing” about which I can have beliefs and learn things. Its “thingness” stems from the internal consistency/compressibility of what the TV is showing.

AI imagines a squirgle.

Similarly, Harry Potter is “a thing”. Like the squirgle, Harry Potter is fictional; there’s no “actual” Harry Potter “out in the real world”, just like the TV doesn’t show any “actual” squirgle “out in the real world”.[1] Nonetheless, I can know things about Harry Potter, the things I know about Harry Potter can have predictive power, and I can discover new things about Harry Potter.

So what does it mean for Harry (or the squirgle) to be "fictional"? Well, it means we can only ever "see" Harry through metaphorical TV screens - be it words on a page, or literal screens.[2]

Claim: human values are “fictional” in that same sense, just like the squirgle or Harry Potter. They’re still “a thing”, we can learn about our values and have beliefs about our values and so forth, but we can only “see” them by looking through a metaphorical screen; they don’t represent some physical thing “out in the real world”. The screen through which we can “see” our values is the reward signals received by our brain.

Background: Value Reinforcement Learning

In a previous post, we presented a puzzle:

Humans sure do seem to have beliefs about our own values and learn about our own values, in a pretty ordinary epistemic way…
But that implies some kind of evidence has to cross the is-ought gap.

We proposed that the puzzle is resolved by our brains treating reward signals as evidence about our own values, and then trying to learn about our values from that reward signal via roughly ordinary epistemic reasoning. The key distinction from standard reinforcement learning is that we have ordinary in...
Momentum of Light in Glass
by Ben — 2024-10-09T20:19:42.088Z

I think that most people underestimate how many scientific mysteries remain, even on questions that sound basic.

My favourite candidate for "the most basic thing that is still unknown" is the momentum carried by light, when it is in a medium (for example, a flash of light in glass or water).

If a block of glass has a refractive index of n, then the light inside that block travels n times slower than the light would in vacuum. But what is the momentum of that light wave in the glass relative to the momentum it would have in vacuum?"

In 1908 Abraham proposed that the light's momentum would be reduced by a factor of n. This makes sense on the surface, n times slower means n times less momentum. This gives a single photon a momentum of ℏω/nc. For ω the angular frequency, c the speed of light and ℏ the reduced Planck constant.

In 1909 Minkowski proposed that actually, the momentum of the same energy worth of light in a material is increased by a factor of n relative to vacuum. This gives a single photon momentum of ℏωn/c.

Which was right? It's still not settled. Not really. A lot of papers will tell you that the "Abraham Minkowski Controversy" is now resolved, but they won't all agree on how it has been resolved. At least three mutually incompatible resolutions have been published. New papers claiming a resolution, or else knocking holes in the old resolutions, are still coming out.

In the rest of this post am going to sketch out why this is a complicated problem, and how some of the proposed solutions work.

An Analogy: The Runner

Why is the momentum of light in glass complicated?

Consider a runner running (as is usual) on land, with some speed c. On their route the runner encounters a region of knee-high water. While travelling through the water our runner travels at a reduced speed of c/n. Now, we ask, what is the momentum of the runner while they are moving through the water?

If we take the mass of everything inside the runner's skin, and multiply it by their velocity then clearly this has fallen by a factor of n relative to their momentum outside the water. However, as the runner wades they will kick a parcel of water along in front of their knees. The total momentum associated with the fact that the runner is in motion is not entirely contained inside their ...
Triangulating My Interpretation of Methods: Black Boxes by Marco J. Nathan
by adamShimi — 2024-10-09T19:13:26.631Z
Link: https://formethods.substack.com/p/triangulating-my-interpretation-of

A perpetual difficulty in my studies is to pinpoint precisely what I’m even trying to do.

That might sounds trivial, but as noted in

a recent post

, if I actually listen to what excites me and interests me and motivates me, I quickly realize that no existing concept fits the weird result, and I find myself in need of engineering a handle for it.

Let’s try something different.

I’m still working through designing this field that I’m dreaming about. Yet I also have a handful of books that are close to it in some way — they capture part of the intuition, part of the goal and approach.

So I’ll triangulate this fabled field through reviewing these books and following what resonates and what doesn’t.

Today, we start with

Black Boxes by Marco J. Nathan

.

Two Aims: Methodology and Philosophy

What is this book even about?

There are two answers, one of which fits with what I care about, and the others which doesn’t.

First, the book is about clarifying and systematizing the method of black boxing, and its place in science. The claim is that many scientists (the main examples are Darwin, Mendel, Skinner, and Friedman[1]) use black boxing (hiding the detail of some complex mechanism of phenomena) for various productive aims, with more or less success. And Nathan wants to dig into this, and try to bring some order to this complexity.

This, I wholeheartedly agree with. What I find fascinating in science and beyond are the methods themselves, in their subtlety and complexity, and I want to systematize and explain them, why they work (or don’t), how they fit with each other, what’s the underlying structure.

The other stated goal of this book is to resolve the conflict between

reductionism and antireductionism

, mostly by dissolving it.

And that, I honestly have no interest in. Don’t misunderstand me: I do agree that the correct approach to most philosophical debate is to dissolve them by showing how they only emerge from wrong assumptions and false dichotomies.[2] Yet this is not something I want to dig in and clarify myself. From my point of view, philosophical debates are almost always void of any relevance to understanding methods and their workings.

Case Studies: Variety and System

Turning to the case studies (which are the beating heart of any good HPS book), the pleasant surprise is their range: as mentioned above, Nathan uses examples from biology (Darwin and Mendel), psychology (...
Scaffolding for "Noticing Metacognition"
by Raemon — 2024-10-09T17:54:13.657Z

tl;dr: I have a more compelling argument for why a generalized "noticing metacognition" skill is useful, than I've previously been able to articulate. And, a simple exercise framework that will probably work decently for most people.

This is basically the parts of Tuning your Cognitive Strategies that were most useful to me, and hopefully a smoother onramp for people who found it confusing.

I'm extremely Noticing-pilled.

I think the ability to notice subtle things in your mind, or in the world around you, is one of the foundational rationality skills. It's necessary for building habits[1], for seeing clues that help you solve confusing problems, for diagnosing how you could have thought that faster, and for generally building a mechanistic model of your mind that lets you become a power-user for your brain.

Despite this, I didn't include Noticing in my recent workshops, because it takes awhile to pay off, and many people initially find it disorienting, or overwhelming, or pointless. Instead of teaching people to "notice their metacognition", I just tell them to do "Livelogging", where you write out your train-of-thought in a google doc while you solve a difficult puzzle, and then afterwards can ask questions like "how could I have thought-those-thoughts faster" and "how can I generalize lessons from that?", while reviewing the Livelog record.

Livelogging is maybe 40% as good as generalized Noticing in the workshop context, but it's enough to get the job done, and requires little training.

But, during last weekend's workshop, I ended up giving a spontaneous lecture about Noticing which I think help motivate it much more clearly. So, if you've been skeptical why you should invest in noticing subtle things, I'm curious if this changes your mind.

Seven cognitive states worth noticing

Ultimately, I think it's worthwhile to gain "generalized noticing", where you have a passive ability to:

a) notice a wide variety of subtle phenomena,
and b) develop a taste for when those phenomena are important enough to actually promote to your attention.

This takes awhile to learn. But there's a particular cluster of cognitive states I find worth paying attention to, which make sense as a unit. And it should make sense why you'd want to notice all of them and do stuff with them.

Moreover, it's relatively straightforward to practice them in the context of solving puzzles, because...