trentmkelly/LessWrong-43k huggingface dataset:
LDT (and everything else) can be irrational
by Christopher King — 2024-11-06T04:05:36.932Z

you should not reject the 'offer' of a field that yields an 'unfair' amount of grain! - Ultimatum Game (Arbital)

In this post, I demonstrate a problem in which there is an agent that outperforms Logical Decision Theory, and show how for any agent you can construct a problem and competing agent that outperforms it. Defining rationality as winning, this means that no agent is rational in every problem.

Symmetrical Ultimatum Game

We consider a slight variation on the ultimatum game to make it completely symmetrical. The symmetrical ultimatum game is a two-player game in which each players says how much money they want. The amount is a positive integer number of dollars. If the sum is ≤$10, both players get the amount of money they choose. Otherwise, they both get nothing.

Now consider the decision problem of playing the symmetrical ultimatum game against a logical decision theorist.

A casual decision theorist does particularly poorly in this problem, since the LDT agent always chooses $9 leaving the casual decision theorist with $1.

How does a LDT agent fare? Well, logical decision theory is still a bit underspecified. However, notice that this question reduces to "how does a LDT agent do against a LDT agent in a symmetrical game?". Without knowing any details about LDT, we must conclude that the expected value is at most $5.

What about a rock with $9 painted on it? The LDT agent in the problem reasons that the best action is to choose $1, so the rock gets $9.

Thus, $9 rock is more rational than LDT in this problem. □

You can't become $9 rock

Now, what makes this problem particularly difficult is how picky the LDT agent in the problem is. If based on the previous you decide to "become $9 rock", the LDT agent will defect against you. If based on the previous section you build a robot that always chooses $9, the LDT agent will defect against that robot. Only a truly natural $9 rock can win.

No agent is rational in every problem

Consider an agent X. There are two cases:

Against $9 rock, X always chooses $1. Consider the problem "symmetrical ultimatum game against X". By symmetry, X on average can get at most $5. But $9 rock always gets $9. So $9 rock is more rational than X.
Against $9 rock, X sometimes chooses more than $1 (thus getting nothing). Consider the problem "symmetrical ultimatum game against $9 rock". X on average gets less than $1. But an agent that alw

...
Join my new subscriber chat
by sarahconstantin — 2024-11-06T02:30:11.059Z

Today I’m announcing a brand new addition to my Substack publication: Rough Diamonds subscriber chat.

This is a conversation space exclusively for subscribers—kind of like a group chat or live hangout. I’ll post questions and updates that come my way, and you can jump into the discussion.

Join chat

How to get started

Get the Substack app by clicking this link or the button below. New chat threads won’t be sent sent via email, so turn on push notifications so you don’t miss conversation as it happens. You can also access chat on the web.

Get app

Open the app and tap the Chat icon. It looks like two bubbles in the bottom bar, and you’ll see a row for my chat inside.

That’s it! Jump into my thread to say hi, and if you have any issues, check out Substack’s FAQ.
Graceful Degradation
by Screwtape — 2024-11-05T23:57:53.362Z

There’s a concept I think about when teaching, which I call Graceful Degradation. The basic idea is, how well does this lesson work if someone doesn’t remember it very well or if I teach it badly?

I picked up Graceful Degradation as an engineer, and you might be familiar with it from that milieu. It's basically the same idea, just used in a different context.

I.

Consider throwing a punch.

Make a fist with your thumb on the outside of your fingers, because you don’t want to pop your thumb. Place your feet shoulder width apart and then take a comfortable step forward with one foot and plant yourself solidly. Bring your fist close to your chest or face, then move it in a straight line from there to your opponent so you waste as little motion as possible. You want to make contact with the first knuckles of your pointer and middle fingers. Don’t hit someone in the face unless you’re wearing boxing gloves, particularly the jaw or mouth since that’s one big crumple zone.

Different martial arts might quibble over some of those details and a good instructor would drill you into much more precise form, but that’s a pretty good crash course on punching people. As a lesson though, it also has the interesting property that every single line is helpful in isolation even if you forget why the line is there.

If you forgot literally everything else except putting your thumb on the outside of your fist, well, at least you’re not going to dislocate your own thumb when you hit someone. If you threw an off-balance hook right to someone’s jaw but remembered to connect with your first two knuckles, that’s still better than landing with the inside of your fist. If you only half remember what your sensei said from years ago, well, you aren’t going to be worse off for using what you do remember. This lesson degrades gracefully.

II.

Compare this to heart transplant.

In a heart transplant,

The patient is put under general anesthetic, drugged so that they are unconscious but still able to breath under their own power.
For many procedures, the patient is put on a heart-lung bypass machine to pump blood through their body while the heart is being operated on. They’re possibly also placed on a ventilator to help their breathing.
The tools, including forceps and a sharp scalpel, are sterilized along with the operating room.
The surgeon cuts a long incision in the patient’s chest and the patient’s r

...
An alternative approach to superbabies
by Towards_Keeperhood — 2024-11-05T22:56:15.740Z

(Note: This post might be slightly funnily written but it is not a joke but serious and important.)

Me: I think it's possible that we might not need to figure out biotechnological advances for how we can create superbabies through embryo selection or so, but that actually they might already be around 1000 or more superbabies born on earth per year, only they happen to be born into an environment where they don't get educated and don't learn the necessary skills to advance science. By superbabies I mean genetic potential for intelligence of >=+7std on the human distribution.

Imaginary person (IP): What, how could that be true? There being some hunter tribe of genetical supergeniuses which are almost not communicating with the rest of humanity?

Me: Sorta yeah.

IP: And where would those superbabies be?

Me: Here:

.

.

.

.

.

IP: Are you joking?

Me: No. See this excellent reddit post (or my wrapper post around it) for why it's IMO a pretty reasonable guess. Actually just read this right now before continuing. It takes only 5min. (ADDED: You could now also read this, but I still strongly recommend reading the reddit post first.)

IP: Ok but it's actually just a fun relatively unlikely possibility right?

Me: No. My current guess would 50% that orcas could do superhuman scientific problem solving (aka >=+7std[1]) if they actually trained themselves at it for human equivalent amounts and had human equivalent interfaces for research (e.g. BCI for using a computer). Though I only tried to form a model of orca intelligence for 2 days[2], so please tell me more evidence and considerations.

Though even if we tried significantly[3] I'd only give it like 15% chance that within the next 30 years, orcas could in one year solve theory-bottlenecked problems for which science would've taken 20 years.[4] (Yeah I know 15% is still ridiculously high considering how crazy the plan sounds.)

E.g. orcas might want to rather do orca stuff, or there might be orca-cultural pressure against spending a large chunk of a day studying, or communicating with orcas turns out to be much harder than my median expectation[5]. (Please comment if you have more considerations on why it might (not) work.)

Can someone please look into this?

I think conditional on orcas being >=+7std intelligent, I'd be about as excited about this as about trying to get human superbabies soon.[6]

I'd guess it mi...
Apply to be a mentor in SPAR!
by agucova — 2024-11-05T21:32:45.797Z

Mentor applications are now open for the Spring 2025 round of the Supervised Program for Alignment Research (SPAR), running from February 10th to May 17th, 2025.

Apply as a mentor

SPAR is a remote-first part-time program that connects mentors and mentees for three-month AI safety and governance research projects. Mentor apps are due Nov 25th, and applications for mentees will run from December 15th to January 8th. If you’re interested in participating as a mentee, you can express interest here, and we’ll reach out once applications are open.

You can find out more about the program here. Starting this round, SPAR is being run by Kairos, a new AI safety fieldbuilding organization.

Should I apply as a mentor?

You might be a good fit to be a SPAR mentor if you are a graduate student, academic, full-time AI safety researcher, independent researcher, or have prior full-time relevant research experience (e.g., MATS, Astra, GovAI fellow, etc.). We’re interested in projects that cover technical AI safety, AI policy and governance, AI strategy, or AI security, and we are able to provide funding for compute costs.

Regarding dedication, we expect most mentors to dedicate 2–12 hours a week, depending on how many mentees they’d like to take in and how much supervision they’re interested in providing. Mentors can decide whether to run the project or not based on the applications they get, making proposals zero-commitment until they accept any mentees.
Going Beyond "immaturity"
by moisentinel — 2024-11-05T20:51:48.162Z

I have been using Linux (EndeavorOS) for a couple of months now. I've had a lot of issues running it smoothly: things don't work out-of-the-box, and you have to put in a lot of time, effort and brain-cells to make things work. And so, by no means it is a perfect OS, but will I ever go back to using Windows? NEVER.

Why? Because, Linux is the superior OS in many ways. It has better memory management, is legally free (the kernel is open-source), has a lot of customisation option (you can do anything), is extremely versatile, has great community support, and so on. If you are a power-user, Linux is for you.

But just a year ago, I believed that switching to Linux would give me all kinds of problems- because I knew that it would be tough- and I ended up sticking to Windows. But now I have gone beyond my fears. I chose to have total control of my PC, instead of being dictated and limited by Microsoft.

This reminds me of what Immanuel Kant said, "It is so easy to be immature."[1] What he meant was, it is so easy to be a subordinate and let others take decisions for you. It is so easy to remain in your own bubble. It is so easy to be guided.

Kant's solution to this problem is "Enlightenment", which he describes as:

“Enlightenment is man's emergence from his self-imposed immaturity. <...> Sapere Aude! [dare to know] 'Have courage to use your own understanding!'—that is the motto of enlightenment.”

Dare to know, and to be wise. Dare to get out of your cage and learn to struggle. People, like me from 2 years before, lack the courage Kant is talking about. This is why using Linux is an act of going beyond immaturity for me.

Of course, using Linux is arduous; I won’t deny that. I have borked my PC 2-3 times in the past 4 months, had so much trouble working with applications, and it is just not that stable.

And you might question: “Why are you using something that creates more problems than solve?”

This is where Albert Camus and Byung-Chul Han come in to a rescue. "The struggle is enough to fill a man's heart," says Camus[2]. Fixing bugs, making things work makes me happy. And I learn a lot in the process; you learn from your mistakes, and that is why it is so powerful.

You don't get that opportunity on Windows; everything is too smooth. Everything is the same; there is no mention of negativity, or the Other. According to Han, this lack ...
Intent alignment as a stepping-stone to value alignment
by Seth Herd — 2024-11-05T20:43:24.950Z

I think Instruction-following AGI is easier and more likely than value aligned AGI, and that this accounts for one major crux of disagreement on alignment difficulty. I got several responses to that piece that didn't dispute that intent alignment is easier, but argued we shouldn't give up on value alignment. I think that's right. Here's another way to frame the value of personal intent alignment: we can use a superintelligent instruction-following AGI to solve full value alignment.

This is different than automated alignment research; it's not hoping tool AI can help with our homework, it's making an AGI smarter than us in every way do our homework for us. It's a longer term plan. Having a superintelligent, largely autonomous entity that just really likes taking instructions from puny humans is counterintuitive, but it seems both logically consistent. And it seems technically achievable on the current trajectory - if we don't screw it up too badly.

Personal, short-term intent alignment (like instruction-following) is safer for early AGI because it includes corrigibility. It allows near-misses. If your AGI did think eliminating humans would be a good way to cure cancer, but it's not powerful enough to make that happen immediately, you'll probably get a chance to say "so what's your plan for that cancer solution?" and "Wait no! Quit working on that plan!" (And that's if you somehow didn't tell it to check with you before acting on big plans).

This type of target really seems to make alignment much easier. See the first linked post, or Max Harms' excellent sequence on corrigibility as a singular (alignment) target (CAST) for a much deeper analysis. An AI that wants to follow directions also wants to respond honestly about its motivations when asked, and to change its goals when told to - because its goals are all subgoals of doing what its principal asks. And this approach doesn't have to "solve ethics" - because it follows the principal's ethics.

And that's the critical flaw; we're still stuck with variable and questionable human ethics. Having humans control AGI is not a permanent solution to the dangers of AGI. Even if the first creators are relatively well-intentioned, eventually someone sociopathic enough will get the reins of a powerful AGI and use it to seize the future.

In this scenario, technical alignment is solved, but most of us die anyway. We die as soon...
Why Recursion Pharmaceuticals abandoned cell painting for brightfield imaging
by Abhishaike Mahajan — 2024-11-05T14:51:41.310Z
Link: https://www.owlposting.com/p/why-recursion-pharmaceuticals-abandoned

Note: thank you to Brita Belli, senior communications manager at Recursion, for connecting me to Charles Baker, a VP at Recursion, who led a lot of this work I’ll discuss here + allowed me to interview him about it! I am not affiliated with Recursion in any capacity.

One more note: a few people have pointed it out that I got the cell painting invention date wrong. It’s actually 2013 from this paper, and not 2016 from this paper. My bad! A few words here have been updated to account for this.

Introduction

At this point, you’d be hard pressed to not have heard of Recursion Pharmaceuticals.

They were one of the first biotechs to apply machine-learning (ML) to the problem of drug discovery back in 2013 and, currently, seem to be the only winners in the area. I use ‘winner’ loosely of course — they have yet to push a drug to market and their recent Phase 2 readouts were disappointing (here is some nuance on the whole subject that most article leave out though).[1] Of course, caveats on this readout (maybe) being an unfairly negative assessment of the company’s capabilities, given that it was Chris Gibson’s — the CEO of Recursion — PhD project from 2013[2], and drugs take time to develop.

Nevertheless, they have done something that very few others in their shoes have: survive for years and not be under active threat of going under.

This is no small feat in a field filled with otherwise promising companies that have failed that deceptively simple smell test, Atomwise being the latest case. I am as unsure as anyone as to whether Recursion’s more grandiose promises, (discussed more in detail by Derek Lowe here) will ever come to pass, but at least they are still around.

The bet of the whole company is the following: take a drug (either from a chemical screening library or dreamt up by a model), wash it over a plate of cells, visually observe how the cells react to it, and repeat this a few billion times across many different cells + many genetic knockouts of those cells (which acts as a model of a disease) + many different drugs. The whole idea is also known as phenomics — the study of the visual phenotype of organisms. In this case, the ‘organism’ is a small set of cells. Once you’ve done that, train a model on those petabytes of cellular images, along with the genetic or chemical perturbation applied.

The Recursion bet is that the resulting model would acquire an...
Winning isn't enough
by JesseClifton — 2024-11-05T11:37:39.486Z

In our jobs as AI safety researchers, we think a lot about what it means to have reasonable beliefs and to make good decisions. This matters because we want to understand how powerful AI systems might behave. It also matters because we ourselves need to know how to make good decisions in light of tremendous uncertainty about how to shape the long-term future.

It seems to us that there is a pervasive feeling in this community that the way to decide which norms of rationality to follow is to pick the ones that win. When it comes to the choice between CDT vs. EDT vs. LDT…, we hear we can simply choose the one that gets the most utility. When we say that perhaps we ought to be imprecise Bayesians, and therefore be clueless about our effects on the long-term future, we hear that imprecise Bayesianism is “outperformed” by other approaches to decision-making.

On the contrary, we think that “winning” or “good performance” offers very little guidance. On any way of making sense of those words, we end up either calling a very wide range of beliefs and decisions “rational”, or reifying an objective that has nothing to do with our terminal goals without some substantive assumptions. We also need to look to non-pragmatic principles — in the context of epistemology, for example, things like the principle of indifference or Occam’s razor. Crucially, this opens the door to being guided by non-(precise-)Bayesian principles.

“Winning” gives little guidance

We’ll use “pragmatic principles” to refer to principles according to which belief-forming or decision-making procedures should “perform well” in some sense. We’ll look at various pragmatic principles and argue that they provide little action-guidance.

Avoiding dominated strategies

First, to review some basic points about common justifications of epistemic and decision-theoretic norms:

A widely-used strategy for arguing for norms of rationality involves avoiding dominated strategies. We can all agree that it’s bad to take a sequence of actions that you’re certain are worse for you than something else.[1] And various arguments take the form: If you don’t conform to particular norms of rationality, you are disposed to act in ways that guarantee that you’re worse off than you could be. A number of arguments for Bayesian epistemology and decision theory — Dutch book arguments; arguments for the axioms of r...
Anthropic - The case for targeted regulation
by anaguma — 2024-11-05T07:07:48.174Z
Link: https://www.anthropic.com/news/the-case-for-targeted-regulation

The first two sections are below:

Increasingly powerful AI systems have the potential to accelerate scientific progress, unlock new medical treatments, and grow the economy. But along with the remarkable new capabilities of these AIs come significant risks. Governments should urgently take action on AI policy in the next eighteen months. The window for proactive risk prevention is closing fast.

Judicious, narrowly-targeted regulation can allow us to get the best of both worlds: realizing the benefits of AI while mitigating the risks. Dragging our feet might lead to the worst of both worlds: poorly-designed, knee-jerk regulation that hampers progress while also failing to be effective at preventing risks.

In this post, we suggest some principles for how governments can meaningfully reduce catastrophic risks while supporting innovation in AI’s thriving scientific and commercial sectors.

Urgency

In the last year, AI systems have grown dramatically better at math, graduate-level reasoning, and computer coding, along with many other capabilities. Inside AI companies, we see continued progress on as-yet undisclosed systems and results. These advances offer many positive applications. But progress in these same broad capabilities also brings with it the potential for destructive applications, either from the misuse of AI in domains such as cybersecurity or biology, or from the accidental or autonomous behavior of the AI system itself.

In the realm of cyber capabilities, models have rapidly advanced on a broad range of coding tasks and cyber offense evaluations. On the SWE-bench software engineering task, models have improved from being able to solve 1.96% of a test set of real-world coding problems (Claude 2, October 2023) to 13.5% (Devin, March 2024) to 49% (Claude 3.5 Sonnet, October 2024). Internally, our Frontier Red Team has found that current models can already assist on a broad range of cyber offense-related tasks, and we expect that the next generation of models—which will be able to plan over long, multi-step tasks—will be even more effective.

On the potential for AI exacerbating CBRN (chemical, biological, radiological, and nuclear) misuses, the UK AI Safety Institute tested a range of models from industry actors (including Anthropic) and concluded that:

...models can be used to obtain expert-level knowledge about biology and chemistry. Fo

...