trentmkelly/LessWrong-43k huggingface dataset:
A "Bitter Lesson" Approach to Aligning AGI and ASI
by RogerDearnaley — 2024-07-06T01:23:22.376Z

TL;DR: I discuss the challenge of aligning AGI/ASI, and outline an extremely simple approach to aligning an LLM: train entirely on a synthetic dataset that always shows the AI acting aligned (even when the humans behave badly), and use a conditional training/inference-time technique to lock the LLM into the AI role.

Epistemic status: To me, this looks like an obvious thing to try. It's conceptually very simple: a vast amount of work is required to actually create the synthetic dataset, but the great majority of that is the sort of work that AI can assist with. I don't see any clear reason why this approach couldn't work, at least for AGI, and perhaps even for ASI, but then we don't know for sure how hard a problem Alignment is. However, if you're proposing any solution to Alignment that's more complicated than this (and most of them are), you should probably have an argument for why this conceptually-simple approach won't work, or won't be sufficient.

If you're not already familiar with it, you should first read Rich Sutton's excellent and influential post The Bitter Lesson. (Even if you are already familiar with it, it's a quick reread, only a page-and-a-half long, and its message is worth remembering.)

Why The Alignment Problem is Hard (In My Opinion)

We have been training LLM-based AIs off enormous web + books + video + etc datasets created by humans, which are full of a vast number of examples of human behavior. We are basically "distilling" human intelligence into these LLMs,[1] teaching them to imitate us. In this process, they become familiar with, understand, and learn to imitate basically all aspects of human behavior — including the many problematic ones for Alignment, such as prejudice, deception, power-seeking, and criminality (and even ones like gluttony and lust that have little practical use for a non-corporal intelligence).

We humans are living beings, the products of evolution, so evolutionary psychology applies to us. While we are a social species, good at cooperating on non-zero-sum games, if you put humans in (what they perceive as) a non-iterated zero-sum situation, they will generally act selfishly for the benefit of themselves and their close genetic relatives, just as evolutionary theory would predict. So the behavioral potentials for deception, power-seeking, criminality etc. are all inherent, evolutionarily adaptive, and thus unsurp...
D&D.Sci: Whom Shall You Call?
by abstractapplic — 2024-07-05T20:53:37.010Z

This is a D&D.Sci scenario: a puzzle where players are given a dataset to analyze and an objective to pursue using information from that dataset.

Intended Difficulty: Average

Good news! After an interminable period of waiting for lawyers to finish arguing over the will, you’ve inherited your great-uncle’s sprawling country estate.

Bad news! As you might expect, the estate is super haunted. And from the results you get from applying your (perfectly reliable, standard-issue) Paranormal Phenomenon Evaluation Kit, several of the twenty-three(!) malevolent entities making the place unlivable seem like they’re on the more troublesome end of the spectral spectrum.

Good news! Your new grounds aren’t in the territory of any specific exorcist group, which means that – unlike the hoi polloi – you get to choose which of the six nearby organisations you want to hire to banish each creature, without running afoul of their intricate web of mutual non-compete agreements. While it’s a given that every exorcist eventually gets their ghost, being able to choose the cheapest still matters: your liquid assets currently total only 70000sp, and you’d really rather have some money left over for throwing lavish parties.

Bad news! Another thing all the exorcists agree on is not quoting prices in advance; apparently, the quantities of resources expended in exorcisms are sufficiently unpredictable that their practitioners uniformly insist on only naming their fees after they’ve done the work.

Good news! You’ve successfully pulled records for all the local exorcist groups detailing what PPEK results they got from each of their targets over the last few years, as well as how much they ended up charging for each job. And because things change so slowly in the exorcist world – and spirits in one territory tend to be no more (or differently) troublesome than those in another – you can be confident that trends deduced from these will apply to the horrors haunting your new home.

Bad news! In addition to their (literally and figuratively) arcane rules about territory and prices, several of the exorcist groups have all-too-human arbitrary constraints: the Spectre Slayers and the Entity Eliminators hate each other to the point that hiring one will cause the other to refuse to work for you, the Phantom Pummelers are too busy to perform more than three exorcisms for you before the start of the social...
[Interim research report] Activation plateaus & sensitive directions in GPT2
by StefanHex — 2024-07-05T17:05:25.631Z

This part-report / part-proposal describes ongoing research, but I'd like to share early results for feedback. I am especially interested in any comment finding mistakes or trivial explanations for these results. I will work on this proposal with a LASR Labs team over the next 3 months. If you are working (or want to work) on something similar I would love to chat!
Experiments and write-up by Stefan, with substantial inspiration and advice from Jake (who doesn’t necessarily endorse every sloppy statement I write). Work produced at Apollo Research.

TL,DR: Toy models of how neural networks compute new features in superposition seem to imply that neural networks that utilize superposition require some form of error correction to avoid interference spiraling out of control. This means small variations along a feature direction shouldn't affect model outputs, which I can test:

Activation plateaus: Real activations should be resistant to small perturbations. There should be a "plateau" in the output as a function of perturbation size.
Sensitive directions: Perturbations towards the direction of a feature should change the model output earlier (at a lower perturbation size) than perturbations into a random direction.

I find that both of these predictions hold; the latter when I operationalize "feature" as the difference between two real model activations. As next steps we are planning to

Test both predictions for SAE features: We have some evidence for the latter by Gurnee (2024) and Lindsey (2024).
Are there different types of SAE features,

atomic and composite

features? Can we get a handle on the total number of features?
If sensitivity-features line up with SAE features, can we find or improve SAE feature directions by finding local optima in sensitivity (similar to how Mack & Turner (2024) find steering vectors)?

My motivation for this project is to get data on computation in superposition, and to get dataset-independent evidence for (SAE-)features.

Core results & discussion

I run two different experiments that test the error correction hypothesis:

Activation Plateaus: A real activation is the center of a plateau, in the sense that perturbing the activation affects the model output less than expected. Concretely: applying random-direction perturbations to an activation generated from a random openwebtext input

...
Minimalist And Maximalist Type Systems
by adamShimi — 2024-07-05T16:25:59.448Z
Link: https://epistemologicalfascinations.substack.com/p/minimalist-and-maximalist-type-systems

I have two completely opposed aesthetic preferences inside me.

On one hand, I love minimalism: doing the maximum with as little as possible. This leads to a fascination with disciplines that have the smallest palette and the largest focus (calligraphy, go, writing, japanese cooking…). More abstractly, it’s an aesthetic of focus, of doing a single thing at a time. When in its thrall, I read a single book at a time for example, and ideally a book that requires deep understanding and analysis.

On the other hand, I love maximalism: doing everything at once in an explosion of parallels and alternatives and colors and sensations. Things like playing complex video games, or watching animation, or eating my family’s Tunisian recipes (abusively spicy!) And at the abstract level, it’s an aesthetic of exploration, of jumping from idea to idea, brainstorming and mixing and matching whatever comes to mind without polishing it too much.

I alternate between these two, often oscillating in the same day, sometimes spending weeks in only one mode.

And lately, I’ve noted this manifesting at the level of

type systems

.

We’ve been over this in the

last couple

of posts

, but the gist is that a type system tries to capture what kind of operations can be done on the value used in a computer program, and to flag incompatible uses of such values before even running the program.

It’s a powerful tool, applied in many settings, and implemented in most programming languages. But like any core idea, you can run with it in wildly different ways.

The ML family of programming languages

embodies the standard (academic) approach to type systems: types as mathematical constructs, and their combinations as logical combinations, following

the Curry-Howard isomorphism

. So using such a system let’s you breathe in the minimalism, the ascetism: you have a handful of basic types, a couple of operators, and you combine them cleanly and transparently and compositionally, such that it’s simple to build complex and yet sensible results from these building blocks.

It’s made so that you can think about it in your head without making massive mistakes, and you can know and even prove straightforwardly what will emerge from your manipulations. But that also makes it often more arid that it needs to be, forcing you to formalize in fundamental maths most of what you want to express. And it means too that you don’t get the...
ML4Good Summer Bootcamps - Applications Open [deadline extended]
by YM — 2024-07-05T13:59:20.518Z

ML4Good bootcamps are 10-day bootcamps focusing on upskilling in technical AI safety, exploring governance, and delving into conceptual topics. ML4Good is a non-profit project and free of charge for the participants. We're seeking motivated individuals with some coding experience who want to make a difference in the field of AI Safety.

💻 Activities:

Peer-coding sessions following a technical curriculum with mentors
Presentations by experts in the field
Review and discussion of AI Safety literature
Personal career advice and mentorship

📅 Dates:

Applications are now open for our UK and Germany bootcamps.

🇬🇧 UK Bootcamp

Apply to the 🇬🇧 UK Bootcamp now

August 31st to September 10th, 2024
Application deadline: July 14th, 2024 [deadline extended]

🇩🇪 Germany Bootcamp

Apply to the 🇩🇪 Germany Bootcamp now

September 23rd to October 3rd, 2024
Application deadline: July 14th, 2024

More camps are being planned - sign up on our website to stay informed about upcoming bootcamps and application deadlines.

🌐 More details on our website:

https://www.ml4good.org/programmes/upcoming-bootcamps
Doomsday Argument and the False Dilemma of Anthropic Reasoning
by Ape in the coat — 2024-07-05T05:38:39.428Z

Doomsday Inference

Can we use probability theory to estimate how many people there will be throughout the whole human history? Sure. We can build a probability model, that takes into account birth rates, possible existential hazards, ways to mitigate them and multiple other factors. Such models tend not to be very precise, so we would have pretty large confidence intervals but, we would still have some estimate.

Hmm... this sounds like a lot of work for not much of a result. Can't we just use the incredible psychic powers of anthropics to circumvent all that, and get a very confident estimate just from the fact that we exist? Consider this:

Suppose that there are two indistinguishable possibilities: a short human history, in which there are only 100 billion people and a long human history, in whichthere are 100 trillion people. You happen to be born among the 6th 10-billion group of people. What should be your credence that the history is short?

As short and long history are a priori indistinguishable and mutually exclusive:

P(Short)=P(Long)=1/2

Assuming that you are a random person among all the people destined to be born:

P(6|Short)=1/10

P(6|Long)=1/10000

According to the Law of Total Probability:

P(6)=P(6|Short)P(Short)+P(6|Long)P(Long)=0.05005

Therefore by Bayes' Theorem:

P(Short|6)=P(6|Short)P(Short)/P(6)>0.999

We should be extremely confident that humanity will have a short history, just by the fact that we exist right now.

This strong update in favor of short history solely due to the knowledge of your birth rank is known as the Doomsday Inference. I remember encountering it for the first time. I immediately felt that it can't be right. Back in the day I didn't have the right lexicon to explain why cognition engines can't produce knowledge this way. I wasn't familiar with the concept of noticing my own confusion. But I've already accustomed myself with several sophisms, and even practiced constructing some myself. So I noticed the familiar feeling of "trickery" that signaled that one of the assumptions is wrong.

I think it took me a couple of minutes to find it. I recommend for everyone to try to do it themselves right now. It's not a difficult problem to begin with, and should be especially easy if you've read and understood my sequence on Sleeping Beauty problem.

.

.

.

.

.

.

.

.

.

Did you do it?

.

.

.

.

.

.

.

.

.

Well, regardless, there will be more time for it. First, let'...
Finding the Wisdom to Build Safe AI
by Gordon Seidoh Worley — 2024-07-04T19:04:16.089Z

We may soon build superintelligent AI. Such AI poses an existential threat to humanity, and all life on Earth, if it is not aligned with our flourishing. Aligning superintelligent AI is likely to be difficult because smarts and values are mostly orthogonal and because Goodhart effects are robust, so we can neither rely on AI to naturally decide to be safe on its own nor can we expect to train it to stay safe. We stand a better chance of creating safe, aligned, superintelligent AI if we create AI that is "wise", in the sense that it knows how to do the right things to achieve desired outcomes and doesn't fall into intellectual or optimization traps.

Unfortunately, I'm not sure how to create wise AI, because I'm not exactly sure what it is to be wise myself. My current, high-level plan for creating wise AI is to first get wiser myself, then help people working on AI safety to get wiser, and finally hope that wise AI safety researchers can create wise, aligned AI that is safe.

For close to a decade now I've been working on getting wiser, and in that time I've figured out a bit of what it is to be wise. I'm starting to work on helping others get wiser by writing a book that explains some useful epistemological insights I picked up between pursuing wisdom and trying to solve a subproblem in AI alignment, and have vague plans for another book that will be more directly focused on the wisdom I've found. I thus far have limited ideas about how to create wise AI, but I'll share my current thinking anyway in the hope that it inspires thoughts for others.

Why would wise AI safety researchers matter?

My theory is that it would be hard for someone to know what's needed to build a wise AI without first being wise themself, or at least having a wiser person to check ideas against. Wisdom clearly isn't sufficient for knowing how to build wise and aligned AI, but it does seem necessary under my assumptions, in the same way that it would be hard to develop a good decision theory for AI if one could not reason for oneself how to maximize expected value in games.

How did I get wiser?

Mostly by practicing Zen Buddhism, but also by studying philosophy, psychology, mathematics, and game theory to help me think about how to build aligned AI.

I started practicing Zen in 2017. I picked Zen with much reluctance after trying many other things that didn't work for me, or worked for a while and...
Libs vs Frameworks, Middle-Level Regularities vs Theories
by adamShimi — 2024-07-04T19:01:59.440Z
Link: https://epistemologicalfascinations.substack.com/p/libs-vs-frameworks-middle-level-regularities

Recently, a friend of mine brought out the essential distinction between

libraries and frameworks

in software engineering.

A library is a small software program that does one thing, does it cleanly, and lets you apply this thing to whatever you want. It lives and dies by

the UNIX philosophy

and the

Keep It Simple Stupid (KISS) principle

. At its heart, a library is the digital equivalent of a hammer.

Whereas a framework comes with a whole approach to solving many problems at once, with a common unified and cleaned abstraction. These days the word gets used mostly for web frameworks like

React

, but it applies whenever the program you use dictates how the full thing must be done (the technical term is

inversion of control

). So a framework is much more like a massive and complex factory that works amazingly well for a specific type of product, defined in a particular way.

At its core, this dichotomy is between low-level simplicity, which is infinitely composable and easier to use but also requires more repeated work to build larger wholes, and complex abstraction which captures some core patterns and intuitions but forces you to follow its rules and constraints.

This also means that a nice small library can easily live for as long as the programming language it uses exists and is in use, with next to no maintenance, whereas frameworks require armies of developers to update and anticipate the needs of their users, constantly refactoring the architecture and abstraction because any change reverberates everywhere, until users start dropping en masse for the next framework.

What’s unexpected is that this split also captures a core distinction in the practice of science, highlighted by the great Hasok Chang in his masterpiece “

Inventing Temperature

”: the distinction between middle-level regularities and high-level theories.

Following Herbert Feigl, Chang distinguishes the empirical laws that simply capture and compress some core phenomena (

Archimedes’ law of the lever

,

Snell-Descartes’ law of refraction

,

the ideal gas law

…), which he calls middle-level regularities, from the complex high-level theories that aim to unify and explain all phenomena in a domain (

Newtonian Mechanics

,

Maxwell’s Electromagnetism

,

Lavoisier’s Chemistry

…)

The latter clearly compresses and simplifies much more, giving you an incredible reach on the moment. But Feigl and Chang’s point is that theories...
The Potential Impossibility of Subjective Death
by VictorLJZ — 2024-07-04T18:17:28.141Z

Epistemic Status: ~70% confidence, hoping to get some feedback.

I'm going to argue that if we accept three controversial (but not outlandish) assumptions to be true, then subjective death (the cessation of conscious experiences identifying themselves as "you") is impossible. I will keep this post as concise as possible.

Here are the three assumptions:

We live in a multiverse (e.g. the MWI or Tegmark's MUH is true).
Consciousness can be simulated computationally.
Identity is "transmitted" through similarity of observer-moments.

Given assumption 1, somewhere "out there" (e.g. in a different Everett branch), there will be worlds that contain "revival simulations" of you (simulations of you being revived and regaining consciousness after your death).

Given assumption 2, these simulations will be conscious and have subjective experience.

Given assumption 3, these revival simulations of you are technically "you" since "you" of this instant is a particular observer-moment that is being computed in a multitude of quantum branches or universes. Your identity is transmitted through the continuity of similar observer-moments since these simulations possess the same memories as you before you died (and hence connect with your last observer moment).

Similar to how the quantum immortality argument says that even though your "reality measure" is reduced after an experiment, you are guaranteed to survive from your perspective because you cannot subjectively experience the branches where you die. Here the argument is that when you die, there will be quantum branches or other universes computing observer-moments of "you" experiencing a revival. Hence you should always expect to be revived after death.

These conclusions sound pretty crazy to me intuitively, but if one accepts the assumptions (whether or not those assumptions are true is outside the scope of this post), these conclusions feel quite straightforward.
Consider the humble rock (or: why the dumb thing kills you)
by pleiotroth — 2024-07-04T13:54:15.593Z

When people think about street-fights and what they should do when they find themselves in the unfortunate position of being in one, they tend to stumble across a pretty concerning thought relatively early on: "What if my attacker has a knife?" . Then they will put loads of cognitive effort into strategies for how to deal with attackers wielding blades. On first glance this makes sense. Knives aren't that uncommon and they are very scary, so it feels pretty dignified to have prepared for such scenarios (I apologize if this anecdote is horribly unrelatable to Statesians). The issue is that –all in all– knife related injuries from brawls or random attacks aren't that common in most settings. Weapons of opportunity (a rock, a brick, a bottle, some piece of metal, anything you can pick up in the moment) are much more common. They are less scary, but everyone has access to them and I've met few people without experience who come up with plans for defending against those before they start thinking about knives. It's not the really scary thing that kills you. It's the minimum viable thing.

When deliberating poisons, people tend to think of the flashy, potent ones. Cyanide, Strychnine, Tetrodotoxin. Anything sufficiently scary with LDs in the low milligrams. The ones that are difficult to defend against and known first and foremost for their toxicity. On first pass this seems reasonable, but the fact that they are scary and hard to defend against means that it is very rare to encounter them. It is staggeringly more likely that you will suffer poisoning from Acetaminophen or the likes. OTC medications, cleaning products, batteries, pesticides, supplements. Poisons which are weak enough to be common. It's not the really scary thing that kills you. It's the minimum viable thing.

My impression is that people in AI safety circles follow a similar pattern of directing most of their attention at the very competent, very scary parts of risk-space, rather than the large parts. Unless I am missing something, it feels pretty clear that the majority of doom-worlds are ones in which we die stupidly. Not by the deft hands of some superintelligent optimizer tiling the universe with its will, but the clumsy ones of a process that is powerful enough to kill a significant chunk of humanity but not smart enough to do anything impressive after that point. Not a schemer but an unstable id...