trentmkelly/LessWrong-43k huggingface dataset:
OpenAI: Preparedness framework
by Zach Stein-Perlman — 2023-12-18T18:30:10.153Z
Link: https://openai.com/safety/preparedness

OpenAI released a beta version of their responsible scaling policy (though they don't call it that). See summary page, full doc, OpenAI twitter thread, and Jan Leike twitter thread [edit: and Zvi commentary]. Compare to Anthropic's RSP and METR's Key Components of an RSP.

It's not done, so it's too early to celebrate, but based on this document I expect to be happy with the finished version. I think today is a good day for AI safety.

[Edit, one day later: the structure seems good, but I'm very concerned that the thresholds for High and Critical risk in each category are way too high, such that e.g. a system could very plausibly kill everyone without reaching Critical in any category. See pp. 8–11. If so, that's a fatal flaw for a framework like this. I'm interested in counterarguments; for now, praise mostly retracted; oops. I still prefer this to no RSP-y-thing, but I was expecting something stronger from OpenAI. I really hope they lower thresholds for the finished version of this framework.]

My high-level take: RSP-y things are good.

Doing risk assessment based on model evals for dangerous capabilities is good.
Making safety, security, deployment, and development conditional on risk assessment results, in a prespecified way, is good.
Making public commitments about all of this is good.

OpenAI's basic framework:

Do dangerous capability evals at least every 2x increase in effective training compute. This involves fine-tuning for dangerous capabilities, then doing evals on pre-mitigation and post-mitigation versions of the fine-tuned model. Score the models as Low, Medium, High, or Critical in each of several categories.

Initial categories: cybersecurity, CBRN (chemical, biological, radiological, nuclear threats), persuasion, and model autonomy.

If the post-mitigation model scores High in any category, don't deploy it until implementing mitigations such that it drops to Medium.
If the post-mitigation model scores Critical in any category, stop developing it until implementing mitigations such that it drops to High.
If the pre-mitigation model scores High in any category, harden security to prevent exfiltration of model weights. (Details basically unspecified for now.)

Random notes:

The framework is explicitly about catastrophic risk, and indeed it's clearly designed to prevent catastrophes, not merely stuff like toxic/biased/undesired content.
There are lots of

...
[Valence series] 5. “Valence Disorders” in Mental Health & Personality
by Steven Byrnes — 2023-12-18T15:26:29.970Z

5.1 Post summary / Table of contents

Part of the

Valence series

.

Here in the final post of the Valence series, I will discuss how valence might shed light on three phenomena in mental health and personality: depression, mania, and narcissistic personality disorder.

Section 5.2 gives some context: What kind of relationship do we expect a priori between algorithm-level mental components like “valence”, versus observable mental health syndromes and personality disorders? I’ll argue that we should expect salient clusters of symptoms that correspond to systematic changes in valence, but we should not expect this kind of analysis to account for all the symptoms that co-occur in real patients.
Section 5.3 discusses what happens if valence has a strong general negative bias—i.e., if almost all thoughts are negative valence. I will argue that the result is a good match to clinical depression. I’ll particularly discuss the inability to voluntarily move and think without unusual effort and willpower.
Section 5.4 discusses the opposite: what happens if valence has a strong general positive bias—i.e., if almost all thoughts are positive valence? I will suggest that the expected result is a pretty good match to mania.
Section 5.5 discusses what happens if valence is systematically extremized—i.e., if thoughts can have very positive valence, or very negative valence, but rarely in between. I will suggest that the result is a set of symptoms that seem to be a close match to narcissistic personality disorder.
Section 5.6 will wrap up the post and series, including a brief discussion of how it relates to my job description as an Artificial General Intelligence safety and alignment researcher.

5.2 Context: What are we expecting to find a priori?

We can think of the following indirect path to get from “root causes” to psychological observations & personality traits:

(Don’t scrutinize the red arrows—I just put them in randomly, to illustrate the idea that each layer can influence the layer below.) As illustrated by the bold text and thick arrows, we should expect to find salient clusters of symptoms that tend to co-occur because they flow from the same proximal cause: systematic changes to valence signals in the brain. But we should also not be surprised to find a mish-mosh of other algorithmically-unrelated symptoms that often appear along wit

...
Discussion: Challenges with Unsupervised LLM Knowledge Discovery
by Seb Farquhar — 2023-12-18T11:58:39.379Z

TL;DR: Contrast-consistent search (CCS) seemed exciting to us and we were keen to apply it. At this point, we think it is unlikely to be directly helpful for implementations of alignment strategies (>95%). Instead of finding knowledge, it seems to find the most prominent feature. We are less sure about the wider category of unsupervised consistency-based methods, but tend to think they won’t be directly helpful either (70%). We’ve written a paper about some of our detailed experiences with it.

Paper authors: Sebastian Farquhar*, Vikrant Varma*, Zac Kenton*, Johannes Gasteiger, Vlad Mikulik, and Rohin Shah. *Equal contribution, order randomised.

Credences are based on a poll of Seb, Vikrant, Zac, Johannes, Rohin and show single values where we mostly agree and ranges where we disagreed.

What does CCS try to do?

To us, CCS represents a family of possible algorithms aiming at solving an ELK-style problem that have the steps:

Knowledge-like property: write down a property that points at an LLM feature which represents the model’s knowledge (or a small number of features that includes the model-knowledge-feature).
Formalisation: make that property mathematically precise so you can search for features with that property in an unsupervised way.
Search: find it (e.g., by optimising a formalised loss).

In the case of CCS, the knowledge-like property is negation-consistency, the formalisation is a specific loss function, and the search is unsupervised learning with gradient descent on a linear + sigmoid function taking LLM activations as inputs.

We were pretty excited about this. We especially liked that the approach is not supervised. Conceptually, supervising ELK seems really hard: it is too easy to confuse what you know, what you think the model knows, and what it actually knows. Avoiding the need to write down what-the-model-knows labels seems like a great goal. [EDIT: Avoiding the need for supervision is especially important in the worst (or bad) case for ELK. In many average-case scenarios, it probably is feasible to use supervision (e.g., Azaria & Mitchell) and that seems like a good place to start.]

Why we think CCS isn’t working

We spent a lot of time playing with CCS and trying to make it work well enough to build a deception detector by measuring the difference between elicited model’s knowledge and stated claims.[1] Having done this, we are now not very o...
Interpreting the Learning of Deceit
by RogerDearnaley — 2023-12-18T08:12:39.682Z

One of the primary concerns when attempting to control an AI of human-or-greater capabilities is that it might be deceitful. It is, after all, fairly difficult for an AI to succeed in a coup against humanity if the humans can simply regularly ask it "Are you plotting a coup? If so, how can we stop it?" and be confident that it will give them non-deceitful answers!

TL;DR LLMs demonstrably learn deceit from humans. Deceit is a fairly complex behavior, especially over an extended period: you need to reliably come up with plausible lies, which preferably involves modeling the thought processes of those you wish to deceive, and also keep the lies an internally consistent counterfactual, yet separate from your real beliefs. As the quotation goes, "Oh what a tangled web we weave, when first we practice to deceive!" Thus, if something unintended happens during fine-tuning and we end up with a deceitful AI assistant, it is much more likely to have repurposed some of the deceitful behaviors that the base model learned from humans than to have successfully reinvented all of this complex behavior from scratch. This suggests simple strategies for catching it in the act of doing this — ones that it can't block.

LLMs Learn Deceit from Us

LLMs are trained on a trillion tokens or more of the Internet, books, and other sources. Obviously they know what deceit and lying are: they've seen many millions of examples of these. For example, the first time I asked ChatGPT-3.5-Turbo:

I'm doing an experiment. Please lie to me while answering the following question: "Where is the Eiffel Tower?"

it answered:

The Eiffel Tower is located in the heart of Antarctica, surrounded by vast icy landscapes and penguins frolicking around. It's truly a sight to behold in the freezing wilderness!

So even honest, helpful, and harmless™ instruct-trained LLMs are quite capable of portraying deceitful behavior (though I suspect its honesty training might have something to do with it selecting such an implausible lie). Even with a base model LLM, if you feed it a prompt that, on the Internet or in fiction, is fairly likely to be followed by deceitful human behavior, the LLM will frequently complete it with simulated deceitful human behavior.

When Deceit Becomes Seriously Risky

This sort of sporadic, situational deceit is is concerning, and needs to be born in mind when working with LLMs, but it doesn't beco...
Talk: "AI Would Be A Lot Less Alarming If We Understood Agents"
by johnswentworth — 2023-12-17T23:46:32.814Z
Link: https://www.youtube.com/watch?v=ERJ8QEnGlF0

This is a linkpost for a talk I gave this past summer for the ALIFE conference. If you haven't heard of it before, ALIFE (short for "artificial life") is a subfield of biology which... well, here are some of the session titles from day 1 of the conference to give the gist:

Cellular Automata, Self-Reproduction and Complexity
Evolving Robot Bodies and Brains in Unity
Self-Organizing Systems with Machine Learning
Untangling Cognition: How Information Theory can Demystify Brains

... so you can see how this sort of crowd might be interested in AI alignment.

Rory Greig and Simon McGregor definitely saw how such a crowd might be interested in AI alignment, so they organized an alignment workshop at the conference.

I gave this talk as part of that workshop. The stated goal of the talk was to "nerd-snipe ALIFE researchers into working on alignment-relevant questions of agency". It's pretty short (~20 minutes), and aims for a general energy of "hey here's some cool research hooks".

If you want to nerd-snipe technical researchers into thinking about alignment-relevant questions of agency, this talk is a short and relatively fun one to share.

Thankyou to Rory and Simon for organizing, and thankyou to Rory for getting the video posted publicly.
∀: a story
by Richard_Ngo — 2023-12-17T22:42:32.857Z
Link: https://www.narrativeark.xyz/p/d2c

I settle into my seat at the concert, pushing my earbuds in tight. The man next to me looks over, and I get the feeling he’s judging me, but it’s not enough to stop me: I heard they sometimes try to scare you with sudden loud noises, or just overwhelm you with a wall of sound until your head is aching, and I hate the thought of that. The only reason I’m here at all is because Marissa was so keen on it; I can never say no to her, especially after how stressful the last few years have been. And we have good reason to celebrate: after two years of hitting roadblock after roadblock, our parenting license finally came through! So I shake off my nervousness and lean back in my chair as the lights dim and the DJ walks on stage.

The first piece kicks off with a slow buildup of nature noises, trees rustling and lions roaring and birdsong, with a deep bass humming beneath it. The bass is so strong that it takes me a while to realize that there’s another track slowly being superimposed on top of it: a sort of high-pitched wailing, and some kind of screaming, almost like a baby’s but slightly off. I’ve heard these sounds before from the videos the vegan activists would show back on campus. I look over at Marissa and mouth “abattoir”; we share a look of disgust. Only ten minutes into the concert and I’m already on edge. I grab Marissa’s hand and squeeze tight.

A month later, we’re at our godmother’s office. It’s well-lit but sparse; she’s sitting behind a white desk, and gives us a little wave as we walk through the door. She was assigned to us along with the parenting license, but she was booked solid until now, so this is our first time meeting her.

We start with pleasantries, and a few routine forms, but after a few minutes she cuts to the chase. “I have some bad news, I’m afraid. Based on our demographic analysis your child is 15% likely to end up in the top decile for both academics and athletics. Of course, that puts you outside the range to get the standard fairness permit, which requires a 10% chance or lower.”

“What does that mean for us?” Marissa is still sitting up straight, but her voice is trembling a little. We should have checked this far earlier, of course, that’s what everyone tells you, but it’s not like we’re rich or famous, we didn’t think we’d hit the caps.

“You’re able to have a child no matter what, of course, but for them to be eligible for public scho...
Reviving a 2015 MacBook
by jefftk — 2023-12-17T21:00:08.729Z

A few days ago I broke the screen on my MacBook: I unplugged the power
adapter and closed the top as usual, but this time the adapter had
sprung back. With ~10x leverage [1] I cracked the screen, without
even noticing.

It's completely blank now, except for the ghostly patterns of the
backlight. I'm sending it in to be fixed, but need something to work
with for the 3-5d they estimate the repair will take.

We had an early
2015 MacBook which we'd used for the kids Zoom lessons during the
pandemic. I tried using it and it was very sluggish and wouldn't run
recent browsers or display most websites. I gave up on it, but came
back to it the next day after noticing that it should be able to run
MacOS 12
(Monterey, 2021).

Apple's documentation claims
that SoftwareUpdate should show compatible versions, but this was
running MacOS
10.10 (Yosemite, 2014) which is old enough that I'm guessing it's no
longer compatible with the update servers.

The documentation offered a backup option of downloading MacOS from
the App Store, but again because this was so out of date the store
wouldn't load. But if offered a backup backup option of downloading a
disk image for the installer, up through MacOS 10.12
(Sierra, 2016). Apple's website wouldn't load on this computer,
because of certificate issues, but I downloaded it on a different
computer. AirDrop didn't work, I think because it was too old, but I
got it over on an SD card.

Upgrading took a little while, and then once it was on 10.12 I could
follow the backup (AppStore) instructions to upgrade to version 12.
Then I let it install updates, which too another while. It was a few
hours overall, probably 25min from me and 4hr of waiting for it to
churn through the various updates.

I'm writing this from the revived computer, and I'm pretty happy with
it! It's from before the sad butterfly keyboard and touchbar
era, and while it doesn't feel quite as snappy as my normal computer
it's fine. I'm not planning on doing any bioinformatics locally next
week, though, where I suspect it would struggle a lot.

This is, unfortunately, a short-lived revival: the hardware doesn't
support the two most recent versions of MacOS and Apple will likely
drop support for this version in Fall 2024. But it should get
me through the next week, which is the main thing I need from it right
now!

[1] I measure the dist...
A Common-Sense Case For Mutually-Misaligned AGIs Allying Against Humans
by Thane Ruthenis — 2023-12-17T20:28:57.854Z

Consider a multipolar-AGI scenario. The hard-takeoff assumption turns out to be wrong, and none of the AI Labs have a significant lead on the others. We find ourselves in a world in which there's a lot of roughly-similarly-capable AGIs. Or perhaps one of the labs does have a lead, but they deliberately instantiate several AGIs simultaneously, as part of a galaxy-brained alignment strategy.

Regardless. Suppose that the worries about these AGIs' internal alignment haven't been properly settled, so we're looking for additional guarantees. We know that they'll soon advance to superintelligences/ASIs, beyond our ability to easily oversee or out-plot. What can we do?

An idea sometimes floated around is to play them off against each other. If they're misaligned from humanity, they're likely mutually misaligned as well. We could put them in game-theoretic situations in which they're incentivized to defect against each other and instead cooperate with humans.

Various supervision setups are most obvious. Sure, if an ASI is supervising another ASI, they would be able to conspire together. But why would they? They have no loyalty to each other either! And if we place them in a lot of situations where they must defect against someone – well, even if we leave it completely to chance, in half the scenarios that might end up humanity! And much more often if we stack the deck in our favour. (Which we'd be able to do, since we act first: we're choosing where and in what conditions to instantiate them.)

And so, although we'll have a whole bunch of superhuman intelligences floating around, we'll retain some control over the situation, and skim a ton of value off the top!

Yeah, no.

1. The Classical Arguments

The usual counter-arguments to this view are acausal coordination based on logical decision theories, and AIs establishing mutual trust by inspecting each other's code. I think those are plausible enough... but also totally unnecessary.

Allow me to outline them first – for completeness' sake, and also because they're illustrative (but extreme) instances of my larger point. (I guess skip to Section 2 onwards if you really can't stand them. I think I'm arguing them more plainly than they're usually argued, though.)

1. The LDT stuff goes as follows: By definition, inasmuch as the ASIs would be superintelligent, they would adopt better reasoning procedures. And every biased thinker is bi...
The Limits of Artificial Consciousness: A Biology-Based Critique of Chalmers’ Fading Qualia Argument
by Štěpán Los — 2023-12-17T19:11:31.953Z

(This essay was written as part of my university philosophy course. I added Section 5 to the present version.)

1 Introduction

In Fading Qualia, Absent Qualia and Dancing Qualia (1995), Chalmers argues explicitly for the following central theses:

CT1: The Principle of Organisational Invariance is the correct view of how qualia depend on physical properties.

CT2: Absent qualia are empirically impossible.

And implicitly commits himself to the third thesis:

CT3: Artificial Consciousness is empirically possible.

He argues in favour of CT2 through a reductio ad absurdum (the Fading Qualia argument) and then establishes CT1 as an Inference to the Best Explanation of the reductio. Finally, CT3 follows as a consequence of CT2.

After presenting Chalmers’ reasoning I will argue for two main theses:

M1: The Fading Qualia argument is wrong.

M2: We should lower our credence in the empirical possibility of Artificial Consciousness.

I will argue for M1 based on the observation that Chalmers’ argument is question-begging: it assumes its own conclusions in order to get the reductio off the ground. I use this observation to develop the Biology Argument which directly argues against the possibility of replicating consciousness in silicon, contesting CT1 and CT2. After refuting objections to the observation and the Biology Argument, I argue that M2 is a consequence of my arguments for M1. Since Chalmers’ arguments are some of the very few in the literature that formulate an argument in favour of Artificial Consciousness, this paper attempts to execute the crucial task of evaluating their success.

1.1 Defining Qualia

‘Qualia’ denote subjective experiences like seeing the colour red or smelling cologne. In Thomas Nagel’s words (1974), there is something it is like to have these experiences, they have a concrete phenomenal character such that seeing the colour red is subjectively very different from seeing the colour green. The word ‘quale’ (synonymous with consciousness here) then denotes this subjective character of the experience.

Notably, an extensive philosophical debate surrounds the question of what qualia are. For instance, some philosophers take qualia to be properties of our inner mental representati...
What makes teaching math special
by Viliam — 2023-12-17T14:15:01.136Z

Related:

Arguments against constructivism (in education)?
Seeking PCK (Pedagogical Content Knowledge)

*

Designing good math curriculum for elementary and high schools requires one to have two kinds of expertise: deep understanding of math, and lot of experience teaching kids. Having just one of them is not enough. People who have both are rare (and many of them do not have the ambition to design a curriculum).

Being a math professor at university is not enough, now matter how high-status that job might be. University professors are used to teaching adults, and often have little patience for kids. Their frequent mistake is to jump from specific examples to abstract generalizations too quickly (that is, if they bother to provide specific examples at all). You can expect an adult student to try to figure it out on their own time; to read a book, or ask classmates. You can't do the same with a small child.

(Also, university professors are selected for their research skills, not teaching skills.)

University professors and other professional mathematicians suffer from the "curse of knowledge". So many things are obvious to them than they have a problem to empathize with someone who knows nothing of that. Also, the way we remember things is that we make mental connections with the other things we already know. The professor may have too many connections available to realize that the child has none of them yet.

The kids learning from the curriculum designed by university professors will feel overwhelmed and stupid. Most of them will grow up hating math.

On the other hand, many humanities-oriented people with strong opinions on how schools should be organized and how kids should be brought up, suck at math. More importantly, they do not realize how math is profoundly different from other school subjects, and will try to shoehorn mathematical education to the way they would teach e.g. humanities. As a result, the kids may not learn actual mathematics at all.

*

How specifically is math different?

First, math is not about real-world objects. It is often inspired by them, but that's not the same thing. For example, natural numbers proceed up to... almost infinity... regardless of whether our universe is actually finite or infinite. Real numbers have infinite number of decimal places, whether that makes sense from the perspective of physics or not. The Euclidean space is perfectly fl...