trentmkelly/LessWrong-43k huggingface dataset:
Eight Magic Lamps
by Richard_Ngo — 2023-10-14T04:10:02.040Z
Link: https://www.narrativeark.xyz/p/eight-magic-lamps

“Give me a lever long enough, and a fulcrum on which to place it, and I shall move the world.” - Archimedes

Aladdin started with nothing; but after a sorcerer tasked him to retrieve a magic lamp, the lamp’s genie granted him wealth and fame. His fortune lasted until the sorcerer stole the lamp, leaving Aladdin ruined. But Aladdin stole it back, and left the sorcerer dead.

That’s the thing about magic lamps: when your future depends on a single point of leverage, triumph and ruin are separated only by a knife’s edge.

Muammar Gaddafi started with nothing; but after joining the Libyan military, he gathered a small cabal of soldiers fed up with King Idris’ regime. Their plans were hastened by rumors of a rival coup: had they waited a week longer, a better-prepared group of conspirators would have seized power instead. But Gaddafi struck first—seizing airports, radio stations, and prominent opponents. King Idris went into exile; the rival conspirators were thrown in prison; and Gaddafi reigned for the next 42 years.

That’s the thing about coups: a decapitating strike can sever a chain of command at its narrowest link, changing the lives of millions overnight.

Humans are social creatures, inspired by stories of struggle and sacrifice. A single life can fuel narratives that persist for millennia. Jesus died in ignominy—yet two thousand years later, his face is worshiped across the world. Muhammad was illiterate, but his proclamations still govern the lives of billions. Marx never stepped on a battlefield, but his advocacy of violent struggle sparked revolutions in two eventual superpowers.

None of them created movements out of nothing. Their teachings would not have spread like wildfire if they weren’t tapping into a deep preexisting current of emotion. But that current never cares about exactly which path it takes—it only cares that it can surge downhill. Whoever bores the first hole in the dam gets to choose its direction, shaping the rivers of culture that form in its wake, their names and teachings adulated for millennia.

That’s the thing about ideology: it allows leaders to channel the spirit of the age to carve their personal mark onto history.

Leo Szilard conceived of the nuclear chain reaction in 1933, the year of Hitler’s rise to power. Over the following years, he hid his knowledge, and worked secretly towards finding an element capable of undergoing a chai...
RSPs are pauses done right
by evhub — 2023-10-14T04:06:02.709Z

COI: I am a research scientist at Anthropic, where I work on

model organisms of misalignment

; I was also involved in the drafting process for

Anthropic’s RSP

. Prior to joining Anthropic, I was a Research Fellow at MIRI for three years.

Thanks to Kate Woolverton, Carson Denison, and Nicholas Schiefer for useful feedback on this post.

Recently, there’s been

a lot of discussion and advocacy

around AI pauses—which, to be clear, I think is great: pause advocacy pushes in the right direction and works to build a good base of public support for x-risk-relevant regulation. Unfortunately, at least in its current form, pause advocacy seems to lack any sort of coherent policy position. Furthermore, what’s especially unfortunate about pause advocacy’s nebulousness—at least in my view—is that there is a very concrete policy proposal out there right now that I think is basically necessary as a first step here, which is the enactment of good

Responsible Scaling Policies

(RSPs). And RSPs could very much live or die right now based on public support.

If you’re not familiar with the concept of an RSP, the central idea of RSPs is evaluation-gated scaling—that is, AI labs can only scale models depending on some set of evaluations that determine whether additional scaling is appropriate.

ARC’s definition

is:

An RSP specifies what level of AI capabilities an AI developer is prepared to handle safely with their current protective measures, and conditions under which it would be too dangerous to continue deploying AI systems and/or scaling up AI capabilities until protective measures improve.

How do we make it to a state where AI goes well?

I want to start by taking a step back and laying out a concrete plan for how we get from where we are right now to a policy regime that is sufficient to prevent AI existential risk.

The most important background here is my “

When can we trust model evaluations?

” post, since knowing the answer to when we can trust evaluations is extremely important for setting up any sort of evaluation-gated scaling. The TL;DR there is that it depends heavily on the type of evaluation:

A capabilities evaluation is defined as “a model evaluation designed to test whether a model could do some task if it were trying to. For example: if the model were actively trying to

autonomously replicate

, would it be capable of doing so?”

With the u

...
Dishonorable Gossip and Going Crazy
by Ben Pace — 2023-10-14T04:00:35.591Z

This is a fairly high-context conversation between me (Ben) and my friend (Ren).

Ren used to work at CFAR, and then ~4.5 years ago moved away to join the Monastic Academy in Vermont. Ren had recently visited me and others in the rationalist scene in the Bay and had written a Facebook post that included her saying she'd heard word that people had been gossiping behind her back about whether she'd "gone nuts", which she said was unwholesome and further said she was angry that people thought it was acceptable behavior. As someone who had done some amount of that, I wanted to talk with Ren about it.

We found a time to have a zoom, then I proposed we move to try talking via written dialogue, a format which we both liked quite a bit.

This is a fairly high-context conversation, and the topic I'm sorry if it doesn't make sense to a bunch of readers.

Thursday, October 12th

Hello! This is me.

Ben Pace

You can type and submit too.

Ben Pace

Hello! This is Ren.

Unreal

Cmd-enter (or ctrl-enter on Windows) will submit your message to the dialogue.

Bringing in from our spoken conversation: The thing that felt most worrying to me was your point that, if you suspected a friend had fallen into a cult or become crazy, then you would be concerned for them and want to support them, yet your sense was that rationalists were quietly coordinating to distance themselves from you, which was a very uncaring and somewhat hostile thing to do.

Ben Pace

Okay, so, I have two initial hypotheses about why people would treat you uncaringly if they thought that you had gone crazy.

The rationalists are an in-group for people who believe true things at the cost of all else. Insofar as you (in their judgment) are trying to have true beliefs then you'll be supported, but if you appear to no longer be doing that then you're "out". This is not totally dissimilar to how a christian parish may accept lots of sinners except for the sin of not believing in God / being another religion.
Roughly speaking the rationalist-people are not actually friends, they're a status hierarchy, and within it you are lucky to have 1-3 friends, and if you fall out of the status hierarchy, then you no longer get the benefits of being in the community. In this version of the story you are actually finding out that you didn't have friends.
A third secret thing that I have not yet thought of.

Thoughts on either of these?

Ben Pace

Imme

...
Disentangling Our Terminal and Instrumental Values
by PeterMcCluskey — 2023-10-14T03:35:23.852Z
Link: https://bayesianinvestor.com/blog/index.php/2023/10/13/disentangling-our-terminal-and-instrumental-values/

Disagreements related to what we value seem to explain maybe 10% of the
disagreements over AI safety. This post will try to explain how I think
about which values I care about perpetuating to the distant future.

Robin Hanson helped to clarify the choices in Which Of Your Origins Are
You?:

The key hard question here is this: what aspects of the causal
influences that lead to you do you now embrace, and which do you
instead reject as "random" errors that you want to cut out? Consider
two extremes.

At one extreme, one could endorse absolutely every random element that
contributed to any prior choice or intuition.

...

At the other extreme, you might see yourself as primarily the result
of natural selection, both of genes and of memes, and see your core
non-random value as that of doing the best you can to continue to
"win" at that game. ... In this view, everything about you that won't
help your descendants be selected in the long run is a random error
that you want to detect and reject.

In other words, the more unique criteria we have about what we want to
preserve into the distant future, the less we should expect to succeed.

Do Humans Have Many Terminal Values?

The Elephant in the Brain shows that
we portray ourselves as having a wide variety of noble values, a large
fraction of which are better explained by simple selfishness.

Self-oriented goals related to wealth, prestige, dominance, safety, and
health can be largely explained as results of us valuing our own
empowerment. The post Empowerment is (almost) All We
Need
expands on this idea.

Shard
Theory
tells us that most human values are context-dependent.

Those considerations lead me to conclude that most human values are
better explained as instrumental values (i.e. subgoals) rather than as
terminal values.

But before I go overboard in that direction, I need to remind myself
that evolution produces messy results that can't be classified as
simply as I'd like. E.g. sex. Evolution intended sex to be an
instrumental value that helps to spread genes.

Evolution implemented sex drives in a way that caused humans to often
treat sex more as a terminal goal. Is sex a genuinely terminal goal? Or
is it a goal that's contingent on certain hormones and sensory organs?
Like most of my values, the answer to this question seems
context-dependent. Transhumanist culture seems to guide me toward
treating sex as an instrumental valu...
Literature On Existential Risk From Atmospheric Contamination?
by Yitz — 2023-10-13T22:27:42.651Z

I'm researching the possibility of global atmospheric contamination as a potential existential risk (say via the deliberate release of some extremely toxic chemical which achieves global dispersion [cf. microplastics for evidence this is possible], etc.), and have been having trouble finding any literature talking about the topic that isn't specifically about either nuclear winter or volcanic eruptions. Anyone know of any relevant literature, or at least have advice on what keywords I should be using to learn more?
How to partition teams to move fast? Debating "low-dimensional cuts"
by Bird Concept — 2023-10-13T21:43:53.067Z

jacobjacob

One of my favorite pastimes is to study historical examples of teams who accomplished ambitious projects effectively. Recently on this topic I wrote the post "A Golden Age of Building? Excerpts and lessons from Empire State, Pentagon, Skunk Works and SpaceX".

In this dialogue, I'd like to discuss a paticular clue I have about making effective teams work: the abstraction of a "low-dimensional interface". I believe this is quite key, but from raising it to you in person, it seems you don't believe that!

So let's go ⚔️

I will make a rambly opening statement, that I then expect to get unpacked as we go. My claim is: in order to design and engineer an artefact, you need to move through a high-dimensional solution space. The more constraints you have, the harder it will be to find something. So you'll need a lot of flexibility in easing and tightening various constraints, and trying different things. (I think people who build stuff will resonate with the experience that changing one part of the system design will often have subtle ramifications in a different part.)

All these design trade-offs happen best inside of a single head, that can hold all the context and trade-offs at the same time, and make design choices that track the dependencies better than can be done between two people.

So, when I say that you should "carve team boundaries at low-dimensional interfaces"; I also mean that you should "carve project ownership / leadership in a way that's maximally inclusive of the high-dimensional dependencies the project owner will have to orient around".

In addition to that, low-dimensional interfaces serve the important function of making it harder for stupid requirements or idiotic decisions to propagate through to other parts of the organisations. (Whereas if you have high-dimensional interfaces, someone might decide on something that'll end up really messing you up!)

kave

My immediate response: needs existence proof. Low-dimensional interfaces seem great, to the extent that you can find them. If you can factor a task into a bunch of subtasks with clear success criteria and no interplay among the dependencies, you're in a fantastic position.

Here are a couple of problems I anticipate:

It's not obvious beforehand how to split the task up. When you initially try, you will fail.
There will be a lot of push from the manager to act as if the low

...
Meta-Regulations
by Sable — 2023-10-13T21:23:42.919Z
Link: https://affablyevil.substack.com/p/meta-regulations

The United States has a lot of regulations:

So does the EU.

Each body has, no matter how you count them, tens of thousands of rules to be followed, across all the domains they are responsible for.

This does not have to be a problem. The world is a complicated place, and the number of fields, industries, subjects, topics, and magisteria a government might rightfully want rules for is large.

That being said, there are problems with laws and regulations, ranging from unintended consequences and ratchet effects to slowing down innovation and preventing the construction of everything from houses and green energy to nuclear power.

The Problems With Regulations And Laws

The Ratchet

A big problem with regulations - and laws in general - are that they tend to function like a ratchet.

Once a law is passed, it’s a law, and the same goes for regulations. They don’t get old, they don’t fade, and they don’t die.

In other words, the number of laws and regulations tends to increase monotonically, including the old, the outdated, and the silly.

This creates two problems:

The pace of innovation and construction of new things slows down over time, as more and more rules and bureaucracy are implemented, and

The body of rules grows until every action breaks some rule, at which point the de facto law becomes the subset of laws which are enforced - a decision made by those who enforce the law, giving them arbitrary power over citizens. In other words, when everyone is guilty of something, anyone can be punished for anything.

The Tech Debt

In software development, there’s a concept called Tech Debt. When a company or developer writes software, they’re often under a deadline or dealing with current problems, and so the software they create is made hastily, to solve the immediate problem, and without sufficient tests.

The result is a pile of software that grows more complicated and difficult to work with over time, as more and more one-off solutions are implemented and decisions to prioritize current crises over future needs are made. The pace of development slows, the code becomes harder to work with or fix, and eventually the whole thing has to be thrown out because it finally breaks in a way that can’t be fixed.

Dealing with Tech Debt is practically a field of software engineering by itself, because it genuinely matters - the livelihood of many companies depends on their software continuing to fu...
Hiring: Lighthaven Events & Venue Lead
by Raemon — 2023-10-13T21:02:33.212Z

Apply for the position here

Lightcone is looking to hire someone to manage Lighthaven, a campus located in Berkeley California. This includes managing our day-to-day maintenance, and running events. You’ll be managing a small team.

We will pay whichever is higher:

30% less than your equivalent industry salary
$120,000

What is Lighthaven?

Lighthaven is a campus located in Berkeley, California. It’s run by Lightcone Infrastructure, and we hope to use this place to make the future of humanity go better than it would go otherwise. (Some major focus areas include “ensuring that AI doesn’t kill everyone this century” and “maintain a culture that can think clearly in a confusing world.”)

In practice, we rent out the place cheaply for projects we think will help the future go better. And, to help make ends meet, we may also rent it out for typical market rates to corporate clients.

Your job is to make sure Lighthaven runs smoothly, efficiently, and is delightful/intellectually-stimulating to event participants. We’re looking for someone who brings ambition and agency to the role.

The role: what will you do?

You would be responsible for:

Leading event preparation, support ops, and teardown
Supporting the experience of Lighthaven visitors and office users
Initiating and overseeing ongoing building maintenance and repairs
Initiating and overseeing building improvements and renovations

In addition, you would work on:

Simplifying, optimizing, automating or deleting all the most annoying maintenance work.
Proactively notice ways in which Lighthaven, and the work and community happening here, is not flourishing, and fix those
Any other things that help with crafting an event running engine, that can gradually run more events, for less cost and team overhead, with happier and more productive participants.

Work culture

Aim for the mission. Lighthaven exists to help humanity survive its

most perilous century

. You take deep pride in your work and in supporting this mission.
Lead by example. You are willing to, and often do, any of the kinds of tasks you delegate. At Lighthaven no work is below anyone. It’s not unusual for the organization CEO to carry couches or plunge toilets at a major event.
Don’t be stuffy. You shun bureaucracy and feel claustrophobic in the endless corridors of the

blankfaced

. You enjoy being around events that surprise you, ente

...
Prediction markets covered in the NYT podcast “Hard Fork”
by Austin Chen — 2023-10-13T18:43:29.644Z
Link: https://www.nytimes.com/2023/10/13/podcasts/hard-fork-israel-x-threads.html

The latest episode of Hard Fork has a 20 min section dedicated to prediction markets! Kevin Roose and Casey Newton go into much more depth in this podcast than in the earlier NYT article, covering the history of prediction markets, the rationalist movement, Google’s internal markets, insider trading, etc. They even talk about embedding prediction markets in the New York Times!

You can listen to the full podcast here (the segment runs from 29:30 to 48:15); a transcript is available there too. Some highlights:

Excerpts

Manifest, LK99 and Manifold

Casey Newton

Kevin, you did some great reporting this week about prediction markets, and it all started at something called Manifest, which — I read that and I thought, well, is this just a festival for men? Tell us about what manifest is.

Kevin Roose

[LAUGHS]: So this is a field trip that I had been planning for a while. This was a very fun and interesting reporting trip to a conference for what they call forecasting nerds, so people who like to predict the future and bet on the future. And this was actually something that came out of an episode that we did several months ago about LK-99. Do you remember this episode?

Casey Newton

Yes, of course.

Kevin Roose

So this was the room-temperature superconductor that a group of scientists in South Korea had claimed to have come up with. And there was this period of maybe a week or two where people were hotly debating whether this was real or not. And we mentioned on the show the existence of something called Manifold Markets, which is a prediction-markets platform where people can go and wager fake money on real-world events. And one of the most popular markets was about LK-99. And it was a way to track, like, what the smart-money people thought was going to happen and whether this prediction of a room-temperature superconductor would pan out. Now, it did not, right? LK-99 did not turn out to be a room-temperature superconductor. But I heard from one of the founders of Manifold Markets who said, if you’re interested in prediction markets, we’re actually having a big conference in a few weeks in Berkeley called Manifest, and you should come report on it. And I thought, well, that sounds like a fun trip.

Casey Newton

Yeah, I’d actually predicted that you were going to go to that, so that was interesting. [KEVIN LAUGHS] So you get there. And sort of describe the scene. Because what you’ve ...
[Paper] All's Fair In Love And Love: Copy Suppression in GPT-2 Small
by CallumMcDougall — 2023-10-13T18:32:02.376Z

This is a accompanying blog post to work done by Callum McDougall, Arthur Conmy and Cody Rushing as part of SERI MATS 2023. The work was mentored by Neel Nanda and Tom McGrath. You can find our full paper at https://arxiv.org/abs/2310.04625. We i) summarize our key results, ii) discuss limitations and future work and iii) list lessons learnt from the project

Key Results

Copy Suppression

In the paper, we define copy suppression as the following algorithm:

If components in earlier layers predict a certain token, and this token appears earlier in the context, the attention head suppresses it.

We show that attention head L10H7 in GPT2-Small (and to a lesser extent L11H10) both perform copy suppression across the whole distribution, and this algorithm explains 76.9% of their behavior.

For example, take the sentence "All's fair in love and war." If we ablate L10H7, the model will incorrectly predict "...love and love." A diagram of this process:

And a few more examples:

Copy Suppression-Preserving Ablation

To test how much of L10H7's behaviour we explained, we designed a form of ablation we called copy suppression preserving ablation (CSPA). The idea is that CSPA should delete all functionality of the head except for the things which are copy suppression. If this completely destroys performance then the head isn't doing copy suppression; if it doesn't affect performance at all.

The technical details of CSPA can be found in our paper. Very loosely, CSPA does two important things:

Delete all information moving from source token S to destination token D, except for those where token S is predicted at D before head L10H7.
For the information which isn't deleted, project it onto the unembedding for token S (and throw away the positive component), so we're only capturing the direct effect of suppressing the token we're attending to.

The key result of this paper was the graph below. DCSPA is the KL divergence between (predictions after applying CSPA) and (original predictions). DMA is the KL divergence between (predictions after mean-ablating) and (original predictions). Each point is the average of a bunch of these values (we've grouped according to the value of DMA). In other words, each (x, y) point on this graph represents (average KL div from mean ablating, average KL div from CSPA) for...