trentmkelly/LessWrong-43k huggingface dataset:
Gorges of gender on a terrain of traits
by dkl9 — 2024-10-22T16:18:29.060Z
Link: https://dkl9.net/essays/gender_gravity.html

Among human populations I'm familiar with, all of these variables of appearance positively correlate:

hair length
chest protrusion
proportion of body fat at hips
negative facial height
absence of facial hair
voice pitch

There are more variables clearly correlated with these. These are just the first that come to mind.

If you see a lot of variables correlating like that, you will infer a hidden variable unifying them. Let's approximate that variable as a weighted average of those traits I listed. Call it "blurk" (a nonce word).

Sith blurk is a weighted average, medium blurk could come from a medium value for each trait. Or medium blurk could come from low values for some and high values for others. One person with long hair, large chest, little hip fat, a tall face with little hair, and a medium-pitch voice would have medium blurk.

It happens that most cisgender men have low blurk, and most cisgender women have high blurk.

Kinda. That diagram suggests that women and men respectively maximise and minimise blurk. In practice, cisgender (henceforth "cis") women cluster around a submaximal-but-high average amount of blurk, and, likewise, cis men around a superminimal-but-low average.

This is building up to a model of how I (and probably some other people) naturally perceive gender. More important than the (perhaps overcomplicated) model itself is the set of concept-handles it implies. How one should perceive gender, or the intrinsic nature of gender, is beyond my qualifications, let alone the scope of this document.

All the variables I listed leading to a blurk-value are superficial and quickly-perceived. Many deeper, less obvious, arguably more important traits also correlate with blurk. But blurk is only useful as an intermediate variable in often-subconscious perception. What matters when defining blurk is what I first see.

Perceiving a person almost always includes perceiving a gender. The usual approximation relies on the two main blurk-clusters, rounding everyone to "woman" or "man" according to the nearest blurk-cluster-centre. But brains tend to think continuously, in spectra, rather than abruptly, in 'chotomies. So the rounding happens more gradually, like balls rolling down into valleys due to gravity.

Seeing a person places a ball along this virtual terrain according to their blurk. The ball then rolls based on local slope. It eventually reaches one of two local mini...
A Defense of Peer Review
by Niko_McCarty — 2024-10-22T16:16:49.982Z
Link: https://www.asimov.press/p/peer-review

This article by Dan Elton (moreisdifferent.blog) was published by Asimov Press.

“I think peer review is, like democracy, bad, but better than anything else."

— Timothy Bates, University of Edinburgh

I used to see peer review as primarily good — an important gatekeeping process, essential for protecting capital “S” Science from all manner of quackery and nonsense. Looking back, I was clearly naive; plenty of junk science gets published in top journals despite the supposed guardrails of peer review. Given the replication crisis, an exponentially growing

deluge

of low-quality or fraudulent work, and

growing evidence

that science is stagnating, some

have

been

wondering

if we should get rid of peer review altogether and try something different.

One prominent essay forcibly arguing against peer review is “

The Rise and Fall of Peer Review

,” by Adam Mastroianni. In it, Mastroianni calls peer review a “failed experiment” and concludes that the best path forward is to do away with it altogether. He suggests that peer review doesn’t “do the thing it’s supposed to do” — namely, “catch bad research and prevent it from being published,” so perhaps scientists would be better off uploading PDFs to the internet where review and oversight can happen post-publication.

1

Mastroianni isn’t exactly sure what a post-peer review world would look like, but

in follow-up

essays

he encourages experimentation. Similar sentiments

have been expressed

elsewhere, such as by Brandon Stell, the founder of

PubPeer

, a website where anyone can leave comments about academic articles.

My experience at the receiving end of peer review has been positive; I’ve co-authored 42 peer-reviewed articles, mostly in the fields of physics and artificial intelligence for healthcare. But despite my good track record with peer review, it is clear that the existing system has serious flaws. We have all heard stories of high-quality work getting rejected while mediocre or error-laden work slips through (numerous

high-profile fraud cases

come to mind). It’s also undeniable that we lack enough data to say whether we get sufficient returns from peer review relative to the enormous amounts of time, resources, and energy we invest in it. Overall, issues such as these are why I found — and continue to find — many of Mastroianni’s arguments compelling.

Still, shortly after I read Mastroianni’s article,

Ben Recht

asked on Twitter...
[Paper Blogpost] When Your AIs Deceive You: Challenges with Partial Observability in RLHF
by Leon Lang — 2024-10-22T13:57:41.125Z
Link: https://arxiv.org/abs/2402.17747

TL;DR

There has been a lot of discussion on Lesswrong on concerns about deceptive AI, much of which has been philosophical. We have now written a paper that proves that deception is one of two failure modes when using RLHF improperly. It's called “

When Your AIs Deceive You: Challenges with Partial Observability in Reinforcement Learning from Human Feedback

”, written with my great co-authors Davis Foote, Stuart Russell, Anca Dragan, Erik Jenner, and Scott Emmons, and has recently been accepted to NeurIPS. We hope that this paper can be a useful pointer for a rigorous discussion of deception caused by RLHF.

Earlier coverage: Scott has discussed the work in other places:

Tweet Thread

AXRP Podcast with Daniel Filan

Talk at the Technical AI Safety Conference, Tokyo

This post is focused more on my own perspective, so I hope it can complement Scott’s earlier coverage!

Brief summary: This is a theoretical paper on what goes wrong when the AI is trained to produce observations that look good to the human, instead of being trained to produce what is actually good. One resulting failure mode is deceptive inflation of the performance: it looks better than it is. The paper also discusses a theoretical procedure for improving RLHF by modeling human beliefs, which seems worthwhile to explore further.

Introduction

Reinforcement Learning from Human Feedback (RLHF) was developed by safety researchers

, including Paul Christiano, Jan Leike, and Dario Amodei. It has a simple underlying idea: If you want an AI that acts according to human preferences, then learn a reward model that predicts the preferences of human evaluators and use it to train your AI. In Paul's view (as I understand it), RLHF effectively serves as a safety baseline that helps to empirically study alignment concerns and start tackling more speculative problems. Recently, variants have been used in ChatGPT, Gemini, and Claude (with Claude using a variant known as constitutional AI), making RLHF very entangled with capability progress over the past few years.

The paper and this post are not meant to discuss the overall impact or outlook of RLHF. Instead, it is mainly about one specific issue (and related concerns):

If the AI can do bad things that look better to the human evaluators than they actually are, RLHF can incentivize the AI to adopt such strategies.

Similar issues have been discussed before:...
[Intuitive self-models] 6. Awakening / Enlightenment / PNSE
by Steven Byrnes — 2024-10-22T13:23:08.836Z

6.1 Post summary / Table of contents

Part of the Intuitive Self-Models series.

I do have meditation experience—in my lifetime, I have probably logged as much as several hours of total time spent meditating! And I was keeping up my meditation practice until as recently as 2007! OK fine, obviously I won’t be speaking from personal experience here. But I will offer some opinions anyway, with pretty low confidence all around.

In

§4.2

I talked about how hard it is to change an intuitive self-model. Well, look to the hardcore meditators if you want to find a bunch of people willing to pour thousands of hours into sculpting their intuitive self-models, like Bernini on clay. My impression is that a whole zoo of different intuitive self-models have come out of this field of inquiry and practice.

I’ll particularly focus on an intuitive self-model called “Persistent Non-Symbolic Experience”[1] (PNSE), a.k.a. “awakening”, “enlightenment”, or (I think?)

kenshō

. Well, it’s probably not “an intuitive self-model” so much as “a category of intuitive self-models”. But I think they have enough overlap for me to make some general comments, trying to “translate” first-person PNSE-related descriptions into legible third-person terms, just as I’ve been doing in the previous four posts, while skipping over a whole ocean of rich details and subtleties.

Needless to say, reading these descriptions is wildly different from experiencing the thing yourself. But hopefully it’s interesting in its own right, and certainly a hell of a lot faster.

I don’t expect this post to be of any practical use in experiencing PNSE, and have no opinion about whether attaining PNSE is even a good idea in the first place—see

§1.3.3

for my general (lack of an) opinion about which intuitive self-models are healthy versus pathological. …OK fine, I put a few pointers to discourse on the pros and cons of PNSE in this expandable box, and a bit more in §6.5.2.2 below.

Pros and cons of pursuing PNSE

I want to point to further reading on both sides of the “pursuing PNSE is a good idea” debate, although pretty please don’t take these as “best and most authoritative arguments” rather than “things that I happen to have randomly stumbled across in the past few weeks”.

IN FAVOR of pursuing PNSE being a good idea is, well, practically everyone who has ever written about their PNSE experiences, e.

...
Resolving von Neumann-Morgenstern Inconsistent Preferences
by niplav — 2024-10-22T11:45:20.915Z

I consider the problem of resolving preferences
that are inconsistent under the von Neumann-Morgenstern
axioms
into consistent preferences. For preferences over deterministic
options, I model inconsistent preferences as directed
graphs,
and the resolution as selecting acyclic
tournaments
with the same vertices and minimal graph-edit
distance,
or Hodge decomposition. For preferences over lotteries, I offer
two different methods for modeling inconsistence and one method
for resolving them: as edge-weighted weakly connected directed
graphs
(resolution via Hodge decomposition) and as arbitrary
relations over
lotteries. None of those two representations prove to be satisfactory. I
apply the findings to propose an algorithm for changing a utility
function as
the underlying set of objects changes.

In economics, decision theory, game theory and parts of artificial
intelligence the standard approach to modeling actors is to assume those
actors have a fixed utility function they optimise Peterson 2017, ch.
6,
Tadelis 2013,
ch. 2,
Russel & Norvig 2010, ch. 16,
following the foundations laid by von Neumann
and Morgenstern von Neumann & Morgenstern 1947, ch.
3.
This model is quite appealing: It assigns a real-numbered value
to each possible outcome, several theorems establish that an
agent with a utility function can't be money-pumped Gustaffson
2022,
and it is compatible with taking Pareto improvements Wald
1947.

However, this model has come under criticism as being
non-descriptive of human preferences, which can be experimentally
shown to violate one or more of the von Neumann-Morgenstern
axioms Allais 1953, El Gamal
2013.
Furthermore, the AI systems humanity has constructed so far usually have
no in-built utility functions and appear inconsistent, as they are often
programs selected by gradient descent to perform well on a loss or reward
function, and it is doubtful that they have internal goal representations
that correspond to the their loss or reward function Hubinger
2019.

This tension between the normative theory of rational agency and the
observations I can make about intelligent systems in the real world
provides a stark contrast and brings up the question of how one could
modify the preferences of intelligent systems to be more consistent.

Motivation

The intuitive case for focusing on resolving inconsistent preferences
is that given we find a normative id...
Lenses of Control
by WillPetillo — 2024-10-22T07:51:06.355Z

This post is a follow-up to

What if Alignment is Not Enough?

, which summarized the core arguments of Substrate Needs Convergence. This post further explores the intuitions and worldview driving this theory, particularly the aspects of it that are the most contentious.

Executive Summary: It is not enough to consider what a system does, but also how its actions affect the world, including how those effects change the system itself, up to the point of stable equilibrium. At the scale of AGI, unintended consequences are inevitable, which gives evolutionary forces room to work, making growth of the system an implied attractor state, eventually overriding conflicting initial goals, such as compatibility with human survival. The only way to be safe from AGI is to not build it.

Terminology

SNC = Substrate Needs Convergence. The theory that AGI will necessarily destroy all biological life as a result of evolution-like pressures.
AGI = Artificial General Intelligence. Highly autonomous systems that outperform humans at most economically valuable work. SNC is primarily concerned about AI at this level.
ASI = Artificial Super Intelligence. AI that possesses the theoretical upper limit of cognitive ability, but is still bounded by physical laws. The fate of the world will likely be sealed before AI reaches this level, but the concept is nonetheless useful as a simplifying abstraction for thought experiments.

Challenging the implicit frame of the super-optimizer

AI safety is a novel field. As such, it is premature to characterize any positive viewpoint as representing a consensus. That said, based on the comments of the prior post in this series and responses to SNC more generally, there seems to be a set of consistent underlying beliefs among the people who engage with SNC enough to comment, but reject the conclusions. I venture to characterize this belief as being that ASI will behave as a persistent super-optimizer. That is, whatever world state is implied by the ASI’s initial starting values is a reasonable approximation of what the world state will be.

The super-optimizer frame admits that the value specification of an ASI is not the sum total of the values it will act on. The theory of Instrumental Convergence suggests that future AI will be motivated by instrumental needs such ...
A Brief Explanation of AI Control
by Aaron_Scher — 2024-10-22T07:00:56.954Z

There are a bunch of great blog posts about the AI Control research direction, but there's not a short explanation/summary that I'm aware of. Here's an attempt at such an explanation, based on my understanding, and aimed at a reader with AISF-level knowledge of AI safety.

We would like to avoid powerful misaligned AI systems from causing catastrophic outcomes. The standard approach to this, the field “AI alignment”, typically focuses on making AI systems aligned in the first place, such that they are not trying to kill you. But we actually care about whether they succeed. AI Control is a subfield of research trying to answer the question, “assuming you have a misaligned AI system, or one that could be misaligned, how can you safely get useful work out of it?”
The background, what situation Control is trying to help with:

We have some AI system that is about as good as human experts on a bunch of tasks that are important for AI development and AI safety research, maybe these are GPT-6, or so. I think it’s a reasonable guess that we’ll get such systems between 2025-2030. We’ll call such a system “GPT-6”.
We do not know how “aligned” GPT-6 is, which is to say, we are unsure if it would try to carry out a coup against humanity if it thought it could. It is “untrusted” ~ “potentially misaligned” ~ “potentially scheming”. We are using it in a setting where humans have control over the model’s inputs, how the outputs are used, the physical servers the model is running on, etc. We are worried that GPT-6 may try to escape this control if given the opportunity, and we institute Control protocols to prevent escape, even if GPT-6s have a high propensity to attempt escape. I’ll use the phrase “escape” to refer to a broad set of highly dangerous and undesirable actions, including a model trying to

exfiltrate its weights

,

hack the datacenter it’s running on

, or

other actions that significantly weaken humanity’s ability to control this AI system

. Escape from human control is likely a key step in executing a coup against humanity, but note that “escape” is sorta being used as a stand-in for some really bad action an AI could attempt. Example escape attempts: inserting a backdoor into code that will later help the AI copy its weights to a remote server, generating AI training data that creates escape-triggering backdoors in t

...
Conversational Signposts—How to stop having boring social interactions
by Declan Molony — 2024-10-22T05:37:56.175Z

I went from being a bad conversationalist to a good one after learning about conversational signposts.

Before defining the concept that led me to having more engaging and diverse social interactions, I’d like to first showcase an awkward chat I had recently.

[The topic of music came up]

Her: “Do you play any instruments?”
Me: “Yeah, I’ve been playing piano for 20 years.”
Her: “Hmm, cool.”

[Awkward silence]

Me: “So, uhh, do you play any instruments? Or are there any instruments you wish you could play?”
Her: “Nah, not really.”

[Awkward silence]

It’s possible she wasn’t interested in talking to me. But it’s also possible that she didn’t know how to advance the conversation. She could have been aided by using…

Conversational Signposts: distinct bits of information that, when followed, lead to divergent paths in a conversation

From the statement “I’ve been playing piano for 20 years,” I can extract out two unique components:

playing piano, and
for 20 years

These are conversational signposts that represent different directions she could have taken our chat.

Her general interest in music could’ve helped guide her to choose which signpost to follow. And if a particular branch of conversation fizzled out, she could’ve always circled back to previous signposts mentioned in the conversation:

Putting my awkward conversation about instruments aside, I want to contrast it with…

A successful example.

While talking to somebody else last week, we seamlessly followed each other’s conversational signposts (which are highlighted in bold).

Me: “Did you get up to anything Friday night?”

Him: “Yeah, I went line dancing at a place called Stony’s.”

Me: [I don’t care about dancing so I opted for a joke.] “Oh cool, does that place double as a weed bar?”

Him: “Huh?”

Me: “Cuz, you know, Stony’s.”

Him: [Groaning]

Me: [Jokingly] “I’ll see myself out” [and fake walked away]. “No, but seriously, is the reason you’re into line dancing because you enjoy the hypnotic effect of dancing in unison? If so, you might be interested in joining a cult—have you ever experimented with that?”

Him: “I think a lot of people experiment in their 20s. Some of the cults I’ve tried were too intense. I remember this one time, I was drinking the blood of a virgin my cult had just sacrificed, and I thought, ‘You know, I think flag football is more my thing.’”

By recognizing and ...
I got dysentery so you don’t have to
by eukaryote — 2024-10-22T04:55:58.422Z
Link: https://eukaryotewritesblog.com/2024/10/21/i-got-dysentery-so-you-dont-have-to/

This summer, I participated in a human challenge trial at the University of Maryland. I spent the days just prior to my 30th birthday sick with shigellosis.

What? Why?

Dysentery is an acute disease in which pathogens attack the intestine. It is most often caused by the bacteria Shigella. It spreads via the fecal-oral route. It requires an astonishingly low number of pathogens to make a person sick – so it spreads quickly, especially in bad hygienic conditions or anywhere water can get tainted with feces.

It kills about 70,000 people a year, 30,000 of whom are children under the age of 5. Almost all of these cases and deaths are among very poor people.

The primary mechanism by which dysentery kills people is dehydration. The person loses fluids to diarrhea and for whatever reason (lack of knowledge, energy, water, etc) cannot regain them sufficiently. Shigella bacteria are increasingly resistant to antibiotics. A disease easily treatable by lots of fluids and antibiotics is becoming more lethal.

Can someone do something?

The deal with human challenge trials

Clinical trials in general are expensive to run but pretty common; clinical trials where you are given the disease – “challenged”, AKA “human challenge trials” – are very rare. The regular way to investigate a possible treatment is to make a study plan, then find people who have the disease and offer to enroll them in the experimental treatment. Challenge trials are less common, but often more valuable for research – shigellosis is a fast-acting disease that is imminently treatable by antibiotics and uncommon in the US. It would be very difficult to test an alternative shigellosis treatment in the US in the conventional way, but it’s a great candidate for challenge trials.

I’d signed up for email alerts on upcoming challenge trials at the nearby University of Maryland, and got one about an upcoming study. It caught my eye that it was for a phage-based treatment. Bacteriophages are really promising antibacterial medicines, not to mention what I’d studied as an undergrad.

Here’s the thing: you really only get good medical research out of human subjects. Also, I could use $4000 and this seemed like a cool way to spend a couple weeks and help out medical research. So I signed up, got a check-in general health appointment, and shortly after, was told I was in. I made plans to spend my 30th birthday in a dysentery ward.

D

...