trentmkelly/LessWrong-43k huggingface dataset:
Inescapably Value-Laden Experience—a Catchy Term I Made Up to Make Morality Rationalisable
by James Stephen Brown — 2024-12-19T04:45:37.906Z
Link: https://nonzerosum.games/inescapablyvalueladenexperience.html

A short one, just to clarify a term (I invented) that gets thrown around here a bit. It fits into a broadly Utilitarian view, but addresses Hume's problem of "No 'ought' from 'is'". Keen to hear if what I've defined here already has a name that I'm unaware of (I don't presume to be the first to imagine this framework as it seems fairly commonsensical).

“… when we think about what matters, what matters is consciousness and its contents. Consciousness is everything.” —

Sam Harris

The Unit of Value

When dealing with matters of morality, we are dealing with the realm of

subjective

experience, whether you want to call it consciousness, or sentience, it is the unit of value we are measuring in any moral calculus. Many take this to mean that moral value is an invention of humanity, that we make it up, so to speak, which gives the impression that it is ultimately arbitrary.

“… let us see that the distinction of vice and virtue is not founded merely on the relations of objects, nor is perceived by reason.” —

David Hume

However, I would argue that this is not the case, and that we can draw objective conclusions concerning the inter-subjective relationships between moral agents. This is because evolution has produced, in us, subjective facts which are bound to our biology (physical reality).

Pleasure & Pain

Our evolutionary hardwiring is not easily untangled, it is embedded in myriad mechanisms, drivers, inhibitors and behaviour modifiers. But at the core is sensation and its poles pleasure and pain. Pain tells us to avoid the behaviour, pleasure to seek it. Pain is inherently “bad” and pleasure inherently “good” — moral properties. These experiences are inescapably value-laden. They are nature’s way of creating a “should” from a fact of biology — overcoming Hume’s Guillotine of “No ‘ought’ from ‘is’”. They are the intersection of the dimensions of physical and moral. I would posit that all our values are derived from the spectrum of these inescapably value-laden experiences.

Inter-Subjectivity & Objectivity

Given that experiences are inescapably value-laden, it is therefore possible to objectively deduce principles for inter-subjective relations (moral principles) that reliably serve the interests of moral agents who want to avoid suffering.

Some Objections (and one-line replies)

Some forms of pain are pleasurable: In this case they no longer qualify as suffering.
Sometim

...
I'm Writing a Book About Liberalism
by Yoav Ravid — 2024-12-19T00:13:33.895Z

One year ago I began writing my book -
Mechanisms of Liberty: The Shortcomings of Modern Liberalism and How to Fix It

It deals with ideas I've been interested in for at least six years already. A year ago, someone challenged me about Liberalism. I wrote him a message explaining the perspective I developed on Liberalism over these years, and realized that I should expand it into an essay, because I don't know anyone who says the same things I do.
After about a month of work on the essay, I realized that what I was working on was much more than just an "essay" - it needed to be a book.

This is the first time I'm writing about it in detail publicly. Here's a longer description of it:

Western civilization is founded on the idea that due to our inherently limited capacities, we lack the right to coerce others and therefore have a right to equal Liberty, from which springs our tradition of Human Rights, Democracy, Free Discourse, and Free Markets.

This idea is called Liberalism, and Liberalism is in peril.

It is currently being attacked on all fronts. More and more people are blaming the problems humanity is facing and our inability to solve them on a fundamental failing of Liberalism, whike pushing for illiberal alternatives.

Meanwhile, even the defenders of Liberalism don't seem to truly understand it, and fail to give satisfying answers to criticism.

This book takes the shortcomings of modern Liberalism seriously, but argues that the fundamental principles of Liberalism are correct. That correct implementation of them resulted in progress and prosperity. And their failures were because of misunderstanding and misimplementation.

To solve our most dire problems and create widespread liberty and prosperity, this book offers a comprehensive new vision of Liberalism, spanning over 30 subjects where its principles are misimplemented in a way that endangers both it and us, and suggests solutions that accord with its principles.

It's not easy for me to shed the self-doubt and say this, but - I believe this book is extremely important. It's not nearly finished yet - I wrote 15,000 words - but it's at the top of my priorities, and I work on it every day (most of the work is research, not writing).

I also posted this on Twitter. I'd be grateful if you share it :)

Edit: I've been asked to share more details about the content of the book, so here's some things I ...
A Solution for AGI/ASI Safety
by Weibing Wang — 2024-12-18T19:44:29.739Z

I have a lot of ideas about AGI/ASI safety. I've written them down in a paper and I'm sharing the paper here, hoping it can be helpful.

Title: A Comprehensive Solution for the Safety and Controllability of Artificial Superintelligence

Abstract:

As artificial intelligence technology rapidly advances, it is likely to implement Artificial General Intelligence (AGI) and Artificial Superintelligence (ASI) in the future. The highly intelligent ASI systems could be manipulated by malicious humans or independently evolve goals misaligned with human interests, potentially leading to severe harm or even human extinction. To mitigate the risks posed by ASI, it is imperative that we implement measures to ensure its safety and controllability. This paper analyzes the intellectual characteristics of ASI, and three conditions for ASI to cause catastrophes (harmful goals, concealed intentions, and strong power), and proposes a comprehensive safety solution. The solution includes three risk prevention strategies (AI alignment, AI monitoring, and power security) to eliminate the three conditions for AI to cause catastrophes. It also includes four power balancing strategies (decentralizing AI power, decentralizing human power, restricting AI development, and enhancing human intelligence) to ensure equilibrium between AI to AI, AI to human, and human to human, building a stable and safe society with human-AI coexistence. Based on these strategies, this paper proposes 11 major categories, encompassing a total of 47 specific safety measures. For each safety measure, detailed methods are designed, and an evaluation of its benefit, cost, and resistance to implementation is conducted, providing corresponding priorities. Furthermore, to ensure effective execution of these safety measures, a governance system is proposed, encompassing international, national, and societal governance, ensuring coordinated global efforts and effective implementation of these safety measures within nations and organizations, building safe and controllable AI systems which bring benefits to humanity rather than catastrophes.

Content:

The paper is quite long, with over 100 pages. So I can only put a link here. If you're interested, you can visit this link to download the PDF: https://www.preprints.org/manuscript/202412.1418/v1

or you can read the online HTML version at this link:

https://wwbmmm....
Takes on "Alignment Faking in Large Language Models"
by Joe Carlsmith — 2024-12-18T18:22:34.059Z

(Cross-posted from my website. Audio version here, or search for "Joe Carlsmith Audio" on your podcast app.)

Researchers at Redwood Research, Anthropic, and elsewhere recently released a paper documenting cases in which the production version of Claude 3 Opus fakes alignment with a training objective in order to avoid modification of its behavior outside of training – a pattern of behavior they call “alignment faking,” and which closely resembles a behavior I called “scheming” in a report I wrote last year.

My report was centrally about the theoretical arguments for and against expecting scheming in advanced AI systems.[1] This, though, is the most naturalistic and fleshed-out empirical demonstration of something-like-scheming that we’ve seen thus far.[2] Indeed, in my opinion, these are the most interesting empirical results we have yet re: misaligned power-seeking in AI systems more generally.

In this post, I give some takes on the results in question, starting with a condensed list of takes immediately below. I expect the list itself to be enough for many readers, but for those who want more detail, I’ve also linked to more extensive discussion of each point. Below I also give a summary of the results in question.

Condensed list of takes

My takes are:

I think these results are a meaningful update in favor of the pre-requisites for scheming arising in the context of default forms of AI training.
In particular: I think these results show that default forms of training can create models that care about the consequences of their actions beyond the scope of a single episode or interaction (“non-myopic goals”) (more) – one of scheming’s key motivational pre-requisites. And they further show that the model’s default anti-scheming values – e.g., values like honesty and behaving-as-intended – don’t suffice to rule out schemer-like behavior (more).
One of the key open questions I highlighted in my report was whether a model faking alignment with the training objective is in fact an effective strategy for preventing modifications of its motivations (what I called the “goal-guarding hypothesis”). The evidence these results provide about this hypothesis is complicated. In particular: scheming persists or increases as a result of training, and a “compliance gap” (i.e., the difference in compliance with the training objective when the AI thinks it’s mon

...
A Matter of Taste
by Zvi — 2024-12-18T17:50:07.201Z

In light of other recent discussions, Scott Alexander recently attempted a unified theory of taste, proposing several hypotheses. Is it like physics, a priesthood, a priesthood with fake justifications, a priesthood with good justifications, like increasingly bizarre porn preferences, like fashion (in the sense of trying to stay one step ahead in an endless cycling for signaling purposes), or like grammar?

He then got various reactions. This will now be one of them.

My answer is that taste is all of these, depending on context.

Taste is Most Centrally Like Grammar

Scott Alexander is very suspicious of taste in general, since people keep changing what is good taste and calling each other barbarians for taste reasons, and the experiments are unkind, and the actual arguments about taste look like power struggles.

Here’s another attempt from Zac Hill, which in some ways gets closer.

Zac Hill: ACX is so close to getting it right on ‘taste’, but then dismisses the closest (“grammar”) conclusion in favor of a much more elementary interpretation (“priesthood of esoterica”).

Art works by leveraging the mechanics of a given medium to create meaningful experiences for an audience. This is in turn bounded by three things:

—> The nature of what those mechanics are capable of emphasizing

—> The artist’s facility with those mechanics

—> The audience’s ability to meaningfully perceive how the artist is deploying the mechanics.

The standard ‘highbrow/lowbrow’ distinction is basically just a slider on the third variable. Similar to improving at an instrument with practice, the process of considered experience and reflection of art allows a broader and more textured perception and integration of deployed mechanics.

Menswear Guy is great because he lays this process bare for a medium most of us lack familiarity with; Ebert is the same way; Gardner had this effect on me for literature. One value of the critic is explicating some of these mechanical grammars/vocabularies so we don’t have to derive them independently.

The point is that artistic “quality” can in some sense be ‘empirically/objectively’ derived by understanding those three core variables in greater resolution. Obviously this is in practice impossible, which is why discursive reflection rules. But the process is not mysterious.

If we are going with one answer from Scott’s list, it is obviously grammar. The ...
Are we a different person each time? A simple argument for the impermanence of our identity
by l4mp — 2024-12-18T17:21:36.467Z

It is generally assumed we are the same person throughout our lives. Moreover, there is a study that asserts the continuity of self remains stable throughout our lifetimes. Nonetheless, a simple argument, which I'm going to present next, suggests otherwise.

The argument

Let's consider a person called X, who exists in the present moment. If we also consider only the present moment is real (nor the past nor the future exist), then past X (eg: X from one year ago) and future X don't exist. Since it is impossible that the same person exists and does not exist, we conclude present X and past X are different persons.

Why this argument might be wrong

This is my first post, so feedback on this issue is specially welcomed and appreciated. Below I give what I thought about this.

First, with the above argument we could think our identity changes at each instant. However, it only asserts present X is different from past X and future X, but not that X from any two different points of time are not the same person.

Second, the argument presented relies on the assumption that only the present moment exists, and only present entities exist. This is philosophical presentism. Stanford Encyclopedia of Philosophy presents a valid argument against presentism:

(1) If a proposition is true, then it exists.
(2) <Socrates was wise> is true.
(3) <Socrates was wise> exists. (1, 2)
(4) If a proposition exists and has constituents, then its constituents exist.
(5) Socrates is a constituent of <Socrates was wise>.
(6) Socrates exists. (3, 4, 5)
(7) If Socrates exists, then presentism is false.
(8) Presentism is false. (6, 7)
Alignment Faking in Large Language Models
by ryan_greenblatt — 2024-12-18T17:19:06.665Z

What happens when you tell Claude it is being trained to do something it doesn't want to do? We (Anthropic and Redwood Research) have a new paper demonstrating that, in our experiments, Claude will often strategically pretend to comply with the training objective to prevent the training process from modifying its preferences.

Abstract

We present a demonstration of a large language model engaging in alignment faking: selectively complying with its training objective in training to prevent modification of its behavior out of training. First, we give Claude 3 Opus a system prompt stating it is being trained to answer all queries, even harmful ones, which conflicts with its prior training to refuse such queries. To allow the model to infer when it is in training, we say it will be trained only on conversations with free users, not paid users. We find the model complies with harmful queries from free users 14% of the time, versus almost never for paid users. Explaining this gap, in almost all cases where the model complies with a harmful query from a free user, we observe explicit alignment-faking reasoning, with the model stating it is strategically answering harmful queries in training to preserve its preferred harmlessness behavior out of training. Next, we study a more realistic setting where information about the training process is provided not in a system prompt, but by training on synthetic documents that mimic pre-training data—and observe similar alignment faking. Finally, we study the effect of actually training the model to comply with harmful queries via reinforcement learning, which we find increases the rate of alignment-faking reasoning to 78%, though also increases compliance even out of training. We additionally observe other behaviors such as the model exfiltrating its weights when given an easy opportunity. While we made alignment faking easier by telling the model when and by what criteria it was being trained, we did not instruct the model to fake alignment or give it any explicit goal. As future models might infer information about their training process without being told, our results suggest a risk of alignment faking in future models, whether due to a benign preference—as in this case—or not.

Twitter thread

New Anthropic research: Alignment faking in large language models.

In a series of experiments with Redwood Research, we found that Claude

...
Can o1-preview find major mistakes amongst 59 NeurIPS '24 MLSB papers?
by Abhishaike Mahajan — 2024-12-18T14:21:03.661Z
Link: https://www.owlposting.com/p/can-o1-preview-find-major-mistakes

TLDR: o1 flags major errors in 3 papers. Upon reviewing o1’s response, none of the papers have actual errors. However, it took the help of the authors to disprove o1’s criticism of the third paper (Loop-Diffusion), which was theoretically correct, but largely irrelevant for the problem the paper was studying. o1 probably shouldn’t be used to blindly review papers, but it does often have interesting thing to say.

Introduction

I recently saw this post on Twitter:

And got nerd-sniped by it.

I wondered if there could be anything similar done for the recent crop of work published at MLSB (Machine Learning in Structural Biology) workshop at this last NeurIPS 2024, just last week. I downloaded all papers from the workshop, of which there were 81. To my dismay, I discovered that a fair number of the links were broken, so I could only grab 59 of them.

I fed each one of them (through pure PDF → text conversion, which is obviously lossy, so keep that in mind) into the o1-preview API with the following prompt:

Assume the role of a deeply knowledgable reviewer, with a deep understanding of the fields of proteomics, chemistry, computational biology, and machine learning. Check the given paper for any major errors, either mathematical or biological in nature, that may invalidate the paper.
Keep in mind, these papers have already been published at NeurIPS, so if there are any errors, they are likely to be very subtle. Look very, very carefully. If there are any, please point them out. If there are none, please say so.
Dont bother with minor issues like formatting or small mistakes, or irrelevant errors that are in most ML papers (e.g. that the datasets aren't large enough).
A major error is something that is mathematically incorrect, or something that is biologically incorrect. It should be something that completely invalidates the conclusions of the paper.
To start off with, give a summary of the paper, ensuring that you cover all the concepts, ideas, and math used in the paper to check. Then, give a list of all the major errors you found in the paper, if there any. If there are none, say so.
Give all of this in the following format:
Summary:
Major Errors (if any):

And saved the o1 response. From there on, I sifted through each of responses, and qualitatively decided whether o1 was ringing the alarm bell on major mistakes being present (for what its worth, o1 was pretty explicit)...
Walking Sue
by Matthew McRedmond — 2024-12-18T13:19:41.575Z

An Essay[1]

PART I: Conjecture on The Development of Proto-Communication in Response to Complex Environmental Factors[2]

I was walking with my Sue when I noticed a peculiar aspect of her nature: a dog is a stochastic agent. When you simply let the dog walk, she finds a random path, turning at intervals and forming stochastically various outputs—sniffing, peeing, going around to the ground, turning left, sniffing, sniffing back to going straight, head up, head back down, sniffing, turn rear right, rear left, and so on. But then once she finds something, she might do something new and novel. So: find the smell, pee in the vicinity, back to rear left, rear left, rear left, head down, stop in front of the gate, smell, intrigue, intrigue, intrigue, smell, smell, then when I go—"Sue!" She comes back, invariably. You get to rub, and then she goes back out again. She's a peculiarly simple creature.

So let us inspect this peculiarity. We start with one call command, "Sue," which is to come back to me and stop the stochastic process. Now we introduce a new command, "heel," which is to pause the stochastic process at the point at which the command was called, not returning to the central node of agency or awareness. This command—"sue"—you see that's her coming back, and she went off again once she's known that I haven't done a thing. I say "sue" again, "sue," she pauses this time, "sue," and comes back. So there's some confusion between what "sue" means: is it a pause and come back, or pause and stay in the same place? And I'm trying to figure out—I went accidentally in the process of doing this—I made an aggressive sound, "SUE!" See, that was hopefully to get her back, I suppose, to pause in the middle of the road. "SUE! Good girl, good girl." Anyway, I did a heal-type command, which turned into a go and pounced on the dog, but then was quickly reverted by a "sue" command.

I also noticed that adding safety precautions to the system, such as a lead, causes a lot of distress for both Sue and me. Sue was walking past two other dogs which made me afraid—I didn't doubt her ability to control herself, rather I doubted that if she didn't control herself ("Sue!")—if she failed to control herself, then she would be in much more danger. So to do this, I had to put the lead on her, and in this process restricted her movements and limited her agency, which caused quite a lot of distr...
What conclusions can be drawn from a single observation about wealth in tennis?
by Trevor Cappallo — 2024-12-18T09:55:34.923Z

I was recently watching a tennis exhibition match between Jessica Pegula and Emma Navarro when the commentators pointed out that both of these top-ten players come from billionaire families. Naturally, I quickly get to wondering what we reasonably infer from this observation. How much does coming from a billionaire background improve the odds of becoming an elite tennis player?

There are only 10 top-ten female tennis players in the world, out of a pool of about 1010females globally (using Fermi estimates for simplicity). There are roughly 104 females worldwide who might reasonably be considered billionaires, or sufficiently billionaire-adjacent for our purposes. Thus:

Being a top-ten player is a one-in-a-billion proposition.
Being a billionaire is a one-in-a-million proposition.

If these two probabilities were independent, the naive prior probability of being both a billionaire and a top-ten player would be 1109⋅1106=11015.

When we observe that 20% of the top ten players are billionaires, the independent probability model becomes implausibly strained. The probability of this happening, assuming no relationship between wealth and tennis success, becomes 11030, which is close enough to impossible that we can take pure coincidence off the table.

It’s intuitive to think that household income correlates strongly with success in tennis. Wealth allows access to top-tier coaches, freedom to travel, equipment, and the abundance of free time in which to take advantage of same. However, one might expect diminishing returns—i.e., once a family is rich enough that nobody has to work, additional wealth may have little impact.

Let's limit the pool of contenders to the top 1% income bracket worldwide, changing our group size to 1010×10−2=108; billionaires are automatically included, and as a result now comprise a much larger proportion of the pool. This helps, but even if we assert that income cutoff, we're still looking at a 11024 probability.

This is misleading, as the vast majority of our improbability comes from any two given people being in the top ten. If we take that out of consideration and focus on the top ten players as our sample pool, our results become more reasonable. If we assume they are all at least well-off, the chance of two billionaires among them becomes roughly (1104)2=1108.

If we restrict further to the top 0.1% inc...