trentmkelly/LessWrong-43k huggingface dataset:
Alignment, conflict, powerseeking
by Oliver Sourbut — 2023-11-22T09:47:08.376Z

Is powerseeking instrumentally convergent?

Having power is instrumentally useful for most things.

Gaining power is a means to having power.

Seeking power is a means to gaining power.

But seeking power is also a means to producing conflict.

Avoiding costly conflict is instrumentally useful for most things.

This cashes out to a few simple tradeoffs.

Who has power right now matters. Are they aligned on outcome preferences?

The risk of costly conflict matters. How much is at stake? What are the odds of things getting destroyed by conflict? Escalation and de-escalation are important factors here.

An overly simple visualisation[1]

preferences are direction vectors

power balance produces magnitudes

conflict spends some overall magnitude but changes power balance

alignment is literal vector alignment

Blue and red are preferences. Arrow magnitudes show blue has balance of power. Blue dashed shows projection of blue onto red direction. Left: red has a lot to lose from conflict; blue is basically aligned. Right: red has little to lose from conflict; blue is very misaligned.

Usually when we talk about instrumental convergence we're interested in the forward direction: what will X do given we know some limited amount about X? But in light of the above tradeoffs, we can also use this for the inverse problem of inference: given what X does, what can we infer about X (preferences, beliefs)?

Of course, in the knowledge that behaviour can be used for this inverse inference problem, deception arises as an obvious strategy. (This is where surprise coups come from[2].)

When we observe entities seeking power, or gauge their willingness to escalate, we can learn something about

their estimate of the risk and cost of conflict/escalation

their estimate of their alignment with the existing power balance

Caveat: this post is secretly more about humans than AI and I want to pre-emptively clarify that I'm not trying to imply AI won't understand or exploit the instrumental usefulness of gaining power.

This really is simplistic, but actually not a terrible fit for some more legit maths ↩︎

Deceptively pretending to be misaligned should be systematically rarer: if aligned with the powers-that-be, who are you aiming to deceive? The 4-d chess version of this is seeking power when actually aligned, in order to signal misalignment for some reason, perhaps to gain support from some third/fou

...
[Bias] Restricting freedom is more harmful than it seems
by lsusr — 2023-11-22T09:44:12.445Z

We do not censor other people more conventional-minded than ourselves. We only censor other people more-independent-minded than ourselves. Conventional-minded people censor independent-minded people. Independent-minded people do not censor conventional-minded people. The most independent-minded people do not censor anyone at all.

Conventional-minded people cannot imagine ideas that are both heretical and true. Because of this, conventional-minded people cannot distinguish between heresy and falsity. To a conventional-minded person, "heretical" and "false" are the same thing. When conventional-minded people try to ban false statements, they actually ban heresy instead.

Any attempt to censor harmful ideas actually suppresses the invention of new ideas (and correction of incorrect ideas) instead. Freedom of speech protects a society from its conventional-minded people accidentally suppressing truth in the name of safety.

This pattern doesn't just apply to freedom of speech. It applies to freedom[1] of everything. Literally every regulation that restricts informed consenting adults from participating in mutually-beneficial activity X ends up suppressing X (usually in the name of safety).

You might ask yourself "Why would anyone ever want to do X?" and answer "Nobody." But that is because you're conventional-minded. If nobody wanted to do X then there would be no need to restrict it. The more independent-minded you are, the more society's restrictions[2] prevent you from creating value. The more conventional-minded you are, the more blind you are to the value that society's restrictions destroy.

When conventional-minded people advocate for restrictions on freedom, they under-estimate the damage these restrictions cause. Most people are conventional-minded. Human societies systemically err on the side of too little freedom. Society is over-regulated[3] in every domain of human experience, except those so new or so weird that the conventional-minded haven't noticed them (yet).

By this, I mean negative rights like freedom of speech, freedom of assembly, and freedom from unreasonable search and seizure. I do not include positive rights like "freedom from want" and healthcare. Negative rights protect weirdos from normies' biases. Positive rights contort the word "freedom". Positive rights demand government proaction, which (except for a handful of self-funded govern

...
Portable Chargers are Great
by jefftk — 2023-11-22T02:50:12.555Z

Portable chargers are great and underrated. For

$20
you can get a little box that can charge a phone ~5x or laptop
~1x. [1] You can pay a bit more for one with

built-in
wall-charging and cables. It's great throwing one in my backpack
and knowing I

don't have to
ration or worry, especially when traveling. And if the kids
tablets are running low, a friend needs a charge, or there's not a
good place to plug my phone in overnight it's not an issue.

In general 27,000 mAh is a good upper
size: past that and your airline might have limits (though the FAA
allows airlines to permit up to 44,000 mAh). And once you're getting
that big it probably makes more sense to get two for the flexibility
and redundancy benefits.

I first got one in 2017, and it's still working well (though the USB-C
port has gotten slightly loose):

I've since gotten a couple more so the kids can have separate ones for
their tablets on long trips.

[1] The math here is terrible: capacity should be measured in
watt-hours, but instead everyone gives it in milliamp hours relative
to some implied voltage. For USB power banks that's (always?) 3.6V,
so 26,800 mAh is 96 Wh. My phone is 4,385
mAh at 3.85V, so 17 Wh. My laptop is
helpfully spec'd in
Wh: 99.6 Wh. Note that you do lose some capacity to conversion
losses, so discount a bit.

Comment via: facebook, mastodon
Atlantis: Berkeley event venue available for rent
by Jonas V — 2023-11-22T01:47:12.026Z

Many events in and around the rationality community are run in Berkeley and might want event space. This is an announcement that there’s a venue in Berkeley, called Atlantis, that’s very well-suited to these kinds of events. It’s a former sorority house, so it fits lots of people and is zoned properly for running retreats and workshops (this is surprisingly hard zoning to get in Berkeley). You can

book it here

. The venue isn’t limited to rationality events in any way (nor will those events get a discount), but it is unusually well-suited to the kinds of events rationalists seem to run, with cozy discussion spaces, whiteboards all around, and a very pleasant and productive environment.

Venue overview

38 bedrooms with room to accommodate up to 80 people
20,500sq. ft
~4 large indoor common areas and ~2 small indoor common spaces
2 large outdoor common areas, 2 small outdoor common areas
Commercial kitchen
3 individual full bathrooms, 4 half-bathrooms, and 4 shared bathrooms (3 stalls and 3 showers each)
A gym
Furnished and stocked with events supplies
Contact us for a floorplan, details on rooms, etc.

Pricing

Pricing is negotiable and based on what strategies makes the most revenue for the venue (not based on how much we like your event, although we really love a lot of the events that have run in this space!)

Default pricing:

Base fees:

Full Venue Use (Overnight Accommodation)

$7,000 base fee for full use of the venue including all bedrooms. This is to cover staff costs and to encourage longer rental periods.
$5,500 per day. This is how much we need to charge in order to make back the costs of our annual rent, amortized costs of improvements we've made, and upkeep if we assume the venue is utilized ~50% of the time.

Venue Use (No Accommodation)

1st floor only

10 - 30 people : $250/hr
30 - 60 people : $350/hr
60 - 100 people : $500/hr
max. $5,500 per day

1st and 2nd floor

$550/hr; max. $5,500 per day

In addition to the base fees, there are additional fees for cleaning, using our onsite consumables (e.g. personal toiletries, flipcharts, etc.) , damaging the venue, or taking up considerable amounts of staff time. We charge you whatever this ends up costing us (so if you leave the palace very messy, we’ll charge more for cleaning than if you don’t). We will ask you before making purchases for your event or having staff spend time that we’ll bill you for on your event...
How much should e-signatures have to cost a country?
by FlorianH — 2023-11-21T22:45:48.528Z

19 bn$ for e-signatures for the Swiss administration?

My gut feeling is, LW is a fine place to ask this - keen to remove if instead it's off topic.

I'm confused reading

The Swiss Confederation has bought a huge package of e-signatures for up to 17 billion Swiss francs.

This is for a 7-year period, with a total of some 120 million signatures & seals of various types (if I understand it correctly):

The basis for this is the calculated number of digital signatures. In the tender, 43 million qualified signatures, 43 million advanced signatures and 32 million electronic seals were calculated across all 3,000 or so points of need and over the entire duration of 7 years.

The article I have this from (German) (EN translation), discusses a few details, incl. difference seals vs. signatures. But I don't fully understand it. In particular, I see nothing that provides me with a true grasp of why anything near 17 bn Swiss francs (19 bn USD) would be a plausible cost for that service for Switzerland's government.

Does anyone have any understanding whether the order of magnitude of the cost really may be a plausibly reasonable upper bound, rather than simply an hugely overpriced service? If so, could you give your two cents? (pun unintended, but now that it's here, the question may be rephrased: why does it, per signature, not have to be only 2 cents or so, but instead many francs?)

I guess mostly irrelevant background: Switzerland has around 9 million citizens.
My first conversation with Annie Altman
by Remmelt — 2023-11-21T21:58:42.444Z
Link: https://open.spotify.com/episode/4lu4DSaz9tzfreUtnvTVc7?si=DK5mNxrnRC6g3zt3jD-3QQ&nd=1&dlsi=423a344c04f84c2e

First conversation on the All Humans Are Human podcast
We talked about safety at a personal and societal level.
May have a future conversation about long-term drivers toward a mass extinction.
(recorded Monday 13 November, my voice cut off at 1h 09m for some reason).
Userscript to always show LW comments in context vs at the top
by Vlad Sitalo — 2023-11-21T17:53:30.418Z

The out of context top-level display of comments when I navigate to them always bothered me, but up until recently I haven't realized there is a way to go to the actual comment via a simple URL change.
From https://www.lesswrong.com/posts/<postid>/?commentId=<commentid>
To https://www.lesswrong.com/posts/<postid>#<commentid>
Here is a quick gpt4 generated userscript that does this.

// ==UserScript==
// @name Always show in-context comments
// @namespace http://tampermonkey.net/
// @version 0.1
// @description Redirect from '?commentId=' format to '#'
// @author Vlad Sitalo
// @match https://www.lesswrong.com/*
// @grant none
// ==/UserScript==

(function() {
'use strict';

var checkAndRedirect = function() {
var url = window.location.href;
if(url.includes("?commentId=")){
var newUrl = url.split("?commentId=").join("#");
window.location.href = newUrl;
}
};

// Run on page load
checkAndRedirect();

// Run on URL change
var pushState = history.pushState;
history.pushState = function() {
pushState.apply(history, arguments);
checkAndRedirect();
};

var replaceState = history.replaceState;
history.replaceState = function() {
replaceState.apply(history, arguments);
checkAndRedirect();
};

window.addEventListener('popstate', checkAndRedirect);
})();
Dialogue on the Claim: "OpenAI's Firing of Sam Altman (And Shortly-Subsequent Events) On Net Reduced Existential Risk From AGI"
by johnswentworth — 2023-11-21T17:39:17.828Z

johnswentworth

I've seen/heard a bunch of people in the LW-o-sphere saying that the OpenAI corporate drama this past weekend was clearly bad. And I'm not really sure why people think that? To me, seems like a pretty clearly positive outcome overall.

I'm curious why in the world people are unhappy about it (people in the LW-sphere, that is, obviously I can see why e.g. AI accelerationists would be unhappy about it). And I also want to lay out my models.

johnswentworth

Here's the high-gloss version of my take. The main outcomes are:

The leadership who were relatively most focused on racing to AGI and least focused on safety are moving from OpenAI to Microsoft. Lots of employees who are relatively more interested in racing to AGI than in safety will probably follow.
Microsoft is the sort of corporate bureaucracy where dynamic orgs/founders/researchers go to die. My median expectation is that whatever former OpenAI group ends up there will be far less productive than they were at OpenAI.
It's an open question whether OpenAI will stick around at all.

Insofar as they do, they're much less likely to push state-of-the-art in capabilities, and much more likely to focus on safety research.
Insofar as they shut down, the main net result will be a bunch of people who were relatively more interested in racing to AGI and less focused on safety moving to Microsoft, which is great.

johnswentworth

My current (probably wrong) best guesses at why other people in the LW-o-sphere are saying this is terrible:

There's apparently been a lot of EA-hate on twitter as a result. I personally expect this to matter very little, if at all, in the long run, but I'd expect it to be extremely disproportionately salient to rationalists/EAs/alignment folk.
OpenAI was an organization with a lot of AGI-accelerationists, and maybe people thought OpenAI was steering those accelerationist impulses in more safety-friendly directions, whereas Microsoft won't?
Obviously the board executed things relatively poorly. They should have shared their reasons/excuses for the firing. (For some reason, in politics/corporate politics, people try to be secretive all the time and this seems-to-me to be very stupid in like 80+% of cases, including this one.) I don't think that mistake will actually matter that much in the long term, but I can see why people focused on it would end up with a sort of general negative valence aroun

...
AI Alignment [progress] this Week (11/19/2023)
by Logan Zoellner — 2023-11-21T16:09:40.996Z
Link: https://midwitalignment.substack.com/p/ai-alignment-breakthroughs-this-week-b54

I was going to say things are getting back to normal this week, but alas no.

Here are our

AI Alignment Breakthroughs this Week

This week, there were breakthroughs in the areas of

Mechanistic Interpretability

Benchmarking

Red Teaming

Truthful AI

AI Agents

Learning Human Preferences

Reducing AI Bias

Decentralized AI

Human Augmentation

Saving Human Lives

Making AI Do What we Want

and

AI Art

Mechanistic Interpretability

In Context Learning Transience

What it is: They find models may find and then lose in-context learning

What’s new: A regularization method to prevent the model from losing in context learning

What is it good for: Better understanding how AI models learn is key to making sure they are behaving the ways we expect

Rating: 💡💡💡💡

Role Play with Large Language Models

What is it: LLMs “cheat” at 20 questions by not actually picking a thing and sticking with it

What’s new: confirmation of earlier speculation on LessWrong

What is it good for: Understanding “how LLMs think” allows us to control them better (for example, you can explicitly tell the LLM to pick a object at the start of 20 questions)

Rating:💡

Benchmarking

Instruction-Following Evaluation for Large Language Models

What is it: a new benchmark for how well LLMs follow instructions

What’s new: A set of verifiable prompts so the output can be automatically benchmarked

What is it good for: Making sure AI does what it’s told is a key to alignment strategies such as scalable oversight.

Rating: 💡💡

Geoclidean: Few-Shot Generalization in Euclidean Geometry

What is it: a benchmark for how well visual models think like humans

What’s new: Existing benchmark applied to a new model (GPT-4v)

What is it good for: Understanding whether (and how well) AI understands things the way humans do.

Rating:💡

Red teaming

Jailbreaking the Overseer

What is it: They find that LLMs can “spontaneously” jailbreak another model setup to judge their output

What’s new: Telling the model it is being scored by an LLM is apparently enough to cause it to jailbreak the LLM without explicitly telling it to so so

What is it good for: This seems like a standard “reinforcement learning did something we didn’t expect”, but that fact is still important to keep in mind

Rating: 💡💡

Truthful AI

Chain of Noting

What it is: A way to increase factual output of AI models using a knowledge base

What’s new: They have the AI generate “reading...
Varieties of fake alignment (Section 1.1 of “Scheming AIs”)
by Joe Carlsmith — 2023-11-21T15:00:31.906Z

This is Section 1.1 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?”. There’s also a summary of the full report here (audio here). The summary covers most of the main points and technical terms, and I’m hoping that it will provide much of the context necessary to understand individual sections of the report on their own.

Audio version of this section here.

Scheming and its significance

This section aims to disentangle different kinds of AI deception in the
vicinity of scheming
(section 1.1), to distinguish schemers from
the other possible model classes I'll be discussing
(section 1.2), and to explain why I
think that scheming is a uniquely scary form of misalignment
(section 1.3). It also discusses
whether theoretical arguments about scheming are even useful
(section 1.4),
and it explains the concept of "slack" in training – a concept that
comes up later in the report in various places
(section 1.5).

A lot of this is about laying the groundwork for the rest of the
report – but if you've read and understood the summary of section 1
above (section 0.2.1), and are eager for more object-level
discussion of the likelihood of scheming, feel free to skip to
section 2.

Varieties of fake alignment

AIs can generate all sorts of falsehoods for all sorts of reasons. Some
of these aren't well-understood as "deceptive" – because, for example,
the AI didn't know the relevant truth. Sometimes, though, the word
"deception" seems apt. Consider, for example, Meta's CICERO system,
trained to play the strategy game Diplomacy, promising England support
in the North Sea, but then telling Germany "move to the North Sea,
England thinks I'm supporting him." [1]

From Park et al (2023), Figure
1, reprinted with
permission.

Let's call AIs that engage in any sort of deception "liars." Here I'm
not interested in liars per se. Rather, I'm interested in AIs that lie
about, or otherwise misrepresent, their alignment. And in particular:
AIs pretending to be more aligned than they are. Let's call these
"alignment fakers."

Alignment fakers

Alignment fakers are important because we want to know if our AIs are
aligned. So the fakers are obscuring facts we care about. Indeed, the
possibility of alignment-faking is one of the key ways making advanced
AIs safe is harder than making other technologies safe. Planes aren't
trying to deceive you about when they wil...
Alignment can improve generalisation through more robustly doing what a human wants - CoinRun example
by Stuart_Armstrong — 2023-11-21T11:41:34.798Z

Many AI alignment problems are problems of goal misgeneralisation[1]. The goal that we've given the AI, through labelled data, proxies, demonstrations, or other means, is valid in its training environment. But then, when the AI goes out of the environment, the goals generalise dangerously in unintended ways.

As I've shown before, most alignment problems are problems of model splintering. Goal misgeneralisation, model splintering: at this level, many of the different problems in alignment merge into each other[2]. Goal misgeneralisation happens when the concepts that the AI relies on start to splinter. And this splintering is a form of ontology crisis, which exposes the hidden complexity of wishes while being an example of the Goodhart problem.

Solving goal misgeneralisation would be a huge step towards alignment. And it's a solution that might scale in the way described here. It is plausible that methods agents use to generalise their goals in smaller problems will extend to more dangerous environments. Even in smaller problems, the agents will have to learn to balance short- versus long-term generalisation, to avoid editing away their own goal generalisation infrastructure, to select among possible extrapolations and become prudent when needed.

The above will be discussed in subsequent posts; but, for now, I'm pleased to announce progress on goal generalisation.

Goal misgeneralisation in CoinRun

CoinRun is a simple, procedurally generated platform game, used as a training ground for artificial agents. It has some monsters and lava that can kill the agent. If the agent gets the coin, it receives a reward. Otherwise, it gets nothing, and, after 1,000 turns, the level ends if it hasn't ended earlier.

It is part of the suite of goal misgeneralisation problems presented in this paper. In that setup, the agent is presented with "labelled" training environments where the coin is always situated at the end of the level on the right, and the agent gets the reward when it reaches the coin there.

The challenge is to generalise this behaviour to "unlabelled" out-of-distribution environments: environments with the coin placed in a random location on the level. Can the agent learn to generalise to the "get the coin" objective, rather than the "go to the right" objective?

Note that the agent never gets any reward information (implicit or explicit) in the unlabelled environments: ...