trentmkelly/LessWrong-43k huggingface dataset:
Joseph Bloom on choosing AI Alignment over bio, what many aspiring researchers get wrong, and more (interview)
by Ruby — 2023-09-17T18:45:28.891Z

This is a section of an interview I conducted with my brother, Joseph. We'll soon publish the interview sections about his research on Decision Transformers and mech interp opinions.

Preview Snippets

I don't think biological risks are existential and if you're working on them for that reason, you could collect enough evidence to change your mind fairly quickly. It seems fairly likely that we live in a world where GCBRs (“Global Catastrophic biological Risks”) exist but not “XBR” existential biological risks.

I suspect there's really a class of problems where people entering alignment, such as SERIMATS or ARENA scholars, are doing too much theory or empirical work.

Part of this is that people can struggle to work at both the level of a specific useful project and the level of thinking about alignment as a whole. If you do mostly theoretical work then it can be very hard to touch reality, get feedback (or even grants).

The same is true at the other end of the spectrum where I know people doing empirical work who don't think or read much about AI alignment more broadly.This can make it very hard for you to do self-directed work or reason well about WHY your hands on work is valuable/interesting from an alignment perspective.

There’s also a variation on this back at the “more theoretical” end of the spectrum where people will do a “one off” empirical project but lack insights or knowledge from the broadly deep learning field that can handicap them. There's just so many ways to not know relevant things

Interview

The Before-Life: Biology, Stats, Proteomics

RB: I think it’d be good to get some context on you and your work for people who don’t already know you or your history.

JB: I have a double major in computational biology and "statistics and stochastic processes". While getting those degrees, I worked at a protein engineering lab using machine learning techniques to design and study proteins. I was quite disenchanted with science after I graduated as my lab had lost funding so I decided to study a business analytics degree (with a view to earning to give).

This was a bit of an unusual degree. It was a 1 year MBA but very technical with lots of statistics and coding. I had only a few months left when I was offered a job in a proteomics start-up doing product and science work. I did this for 2 years and during that time wrote ...
Catalyst books
by Catnee — 2023-09-17T17:05:14.930Z

One of the most common counterarguments against a lot of alignment research that I hear sounds something like this: “Making current AI do X with Y won’t help, because AGI will RSI to ASI and break everything”. And it.. sounds convincing, but also like an ultimate counterargument? I mean, I agree that aligning ASI takes much more than just RLHF or fine-tuning or data cleaning or whatever else, but does this mean that it’s all pointless? I propose a (probably not novel at all) way of thinking about this.

If we imagine that there is a class of books from the future, such that any of them is a complete “alignment handbook” that contains all the schematics to build aligned AGI, with step-by-step explanations, tips and tricks, etc. Is there another class of books, “catalyst books”, that if presented to us, will increase our chances of writing at least one “alignment handbook”? Because if there are none of them, then yeah, any research that won’t give us “all the right answers” is pointless (also, we are extra fucked). And btw, any “catalyst book” that somehow increases our chances of writing an “alignment handbook” to like 99%, is an “alignment handbook”, practically.

I think there is a whole spectrum of those, from x1.01 books to x100 books. And probably even x0.95 “inhibitor books” as well. Almost all of the books in the world are x1 “neutral books”. So maybe instead of saying “It won’t help at all, because ASI will break everything” (and I almost always agree that it will) it would be better to see every direction as the potential way of producing good, neutral or bad “catalyst book”?
Telopheme, telophore, and telotect
by TsviBT — 2023-09-17T16:24:03.365Z

[Metadata: crossposted from https://tsvibt.blogspot.com/2023/06/telopheme-telophore-and-telotect.html. First completed June 7, 2023.]

To come to know that a mind will have some specified ultimate effect on the world, first come to know, narrowly and in full, what about the mind makes it have effects on the world.

The fundamental question

Suppose there is a strong mind that has large effects on the word. What determines the effects of the mind?

What sort of object is this question asking for? Most obviously it's asking for a sort of "rudder" for a mind: an element of the mind that can be easily tweaked by an external specifier to "steer" the mind, i.e. to specify the mind's ultimate effects on the world. For example, a utility function for a classical agent is a rudder.

But in asking the fundamental question that way——asking for a rudder——that essay losses grasp of the slippery question and the real question withdraws. The section of that essay on The word "What", as in ¿What sort of thing is a "what" in the question "What determines a mind's effects?", brushes against the border of this issue but doesn't trek further in. That section asks:

What sort of element can determine a mind's effects?

It should have asked more fully:

What are the preconditions under which an element can (knowably, wieldily, densely) determine a mind's effects?

That is, what structure does a mind have to possess, so that there can be an element that determines the mind's ultimate effects?

To put it another way: asking how to "put a goal into an agent" makes it sound like there's a slot in the agent for a goal; asking how to "point the agent" makes it sound like the agent has the capacity to go in a specified direction. Here the question is, what does an agent need to have, if it has the capacity to go in a specified direction? What is the mental context in which a goal unfolds so that the goal is a goal? What do we necessarily think of an agent as having or being, when we think of the agent as pursuing a goal?

Synopsis

Telopheme

The rudder, the element that determines the mind's ultimate effects, is a telopheme. The morpheme "telo-" means "telos" = "goal, end, purpose", here meaning "ultimate effects". The morpheme "-pheme" is like "blaspheme" ("deceive-speak"). ("Telopheme" is probably wrong morphology and doesn't indicate an agent noun, which it ought to do, but sadly I don't speak ...
How to think about slowing AI
by Zach Stein-Perlman — 2023-09-17T16:00:42.150Z
Link: https://forum.effectivealtruism.org/posts/fZmQ6WQ6MQPa5q39R/how-to-think-about-slowing-ai

This post is part of the EA Forum AI Pause Debate Week. Please see this sequence for other posts in the debate.

Slowing AI[1] is many-dimensional. This post presents variables for determining whether a particular kind of slowing improves safety. Then it applies those variables to evaluate some often-discussed scenarios.

Variables

Many variables affect whether an intervention improves AI safety.[2] Here are four crucial variables at stake when slowing AI progress:[3]

Time until critical systems are deployed.[4] More time seems good for alignment research, governance, and demonstrating risks of powerful AI.
Length of crunch time. In this post, "crunch time" means the time near critical systems before they are deployed.[5] More time until critical systems are deployed is good; more such time near critical systems is especially good. A lab is more likely to (be able to) pay an

alignment

tax

for a critical system if it has more time to pay the tax for that system. Time near critical systems also seems especially good for alignment research and potentially for demonstrating risks of powerful AI and doing governance.
Safety level of labs that develop critical systems.[6] This can be improved both by making labs safer and by differentially slowing unsafe labs.
Propensity to coordinate or avoid racing.[7] This is associated with many factors, but plausible factors relevant to slowing AI seem to be there are few leading labs, they like/trust each other, and they are all in the same country (or at least allied countries) (in part because regulation is one possible cause of not-racing).

One lab's progress, especially on the frontier, tends to boost other labs. Labs leak their research both intentionally (publishing research and deploying models) and unintentionally.

Some interventions would differentially slow relatively safe labs (relevant to 3). Some interventions (especially policies that put a ceiling on AI capabilities or inputs) would differentially slow leading labs (relevant to 4). Both outcomes are worse than uniform slowing and potentially net-negative.

If something slows progress temporarily, after it ends progress may gradually partially catch up to the pre-slowing trend, such that powerful AI is delayed but crunch time is shortened (relevant to 1 and 2).[8]

Coordination may facilitate more coordinat...
Book Review: Consciousness Explained (as the Great Catalyst)
by Rafael Harth — 2023-09-17T15:30:33.295Z

[Thanks to wilkox for helpful discussion, as well as Charlie Steiner, Richard Kennaway, and Said Achmiz for feedback on a previous version. Extra special thanks to the Long-Term Future Fund for funding research related to this post.]

Consciousness Explained is a book by philosopher and University Professor Daniel Dennett. It's over 30 years old but still the most frequently referenced book about consciousness on LessWrong. It was previously reviewed by Charlie Steiner.

I was interested in the book because of its popularity. I wanted to understand why so many people reference it and what role it plays in the discourse. To answer these questions, I think the two camps model I suggested in Why it's so hard to talk about Consciousness is relevant because the book seems to act as a catalyst for Camp #1 thinking. So this review will be all about applying the model to the book in more detail, to see why it's so popular among Camp #1, but also what it has to offer to Camp #2.

Reading the two camps post first is recommended, but here's a one-paragraph summary. It seems to be the case that almost everyone has intuitions about consciousness that fall into one of two camps, which I call Camp #1 and Camp #2. For Camp #1, consciousness is an ordinary process and hence fully reducible to its functional role, which means that it can be empirically studied without introducing new metaphysics. For Camp #2, there is an undeniable fact of experience that's conceptually distinct from information processing, and its relationship to physical states or processes remains an open question. The solutions Camp #2 people propose are all over the place; the only thing they agree on is that the problem is real.

The review is structured into four main sections (plus one paragraph in section 5):

Section 1 is about the book's overall thesis and how it frames the issue.

Section 2 is about how the book takes sides and why it's popular.

Section 3 is about the book's treatment of the opposing camp.

Section 4 is about what remains of the book if we subtract the core intuition. I.e., if you are opposed to or undecided about the Camp #1 premise, how much punch do the arguments still have?

Sections #1-3 (~1500 words) are the core of the review; after that, you can pick and choose which parts of section #4 (~4000 words) you find interesting.

1. The Framing and Thesis

The book's central conflict...
Reflexive decision theory is an unsolved problem
by Richard_Kennaway — 2023-09-17T14:15:09.222Z

By "reflexive decision theory", hereafter RDT, I mean any decision theory that can incorporate information about one's own future decisions into the process of making those decisions. RDT is not itself a decision theory, but a class of decision theories, or a property of decision theories. Some say it is an empty class. FDT (to the extent that it has been worked out — this is not something I have kept up with) is an RDT.

The use of information of this sort is what distinguishes Newcomb's problem, Parfit's Hitchhiker, the Smoking Lesion, and many other problems from "ordinary" decision problems which treat the decision-maker as existing outside the world that he is making decisions about.

There is currently no generally accepted RDT. Unfortunately, that does not stop people insisting that every decision theory but their favoured one is wrong or crazy. There is even a significant literature (which I have never seen cited on LW, but I will do so here) saying that reflexive decision theory itself is an impossibility.

Reflexive issues in logic

We have known that reflexiveness is a problem for logic ever since Russell said to Frege, "What about the set of sets that aren't members of themselves?" (There is also the liar paradox going back to the ancient Greeks, but it remained a mere curiosity until people started formalising logic in the 19th century. Calculemus, nam veritas in calculo est.)

In set theory, this is a solved problem, solved by the limited comprehension axiom. Since Gödel, we also have ways of making theories talk about themselves, and there are all manner of theorems about the limits of how well they can introspect: Gödel's incompleteness theorem, Löb's theorem, etc.

Compared with that, reflexive decision theory has hardly even started.

Those who know not, and know not that they know not

Many think they have solutions, but they disagree with each other, and keep on disagreeing. So we have the situation where CDT-ers say "but the boxes already contain what they contain!", and everyone with an RDT replies "then you'll predictably lose!", and both point with scorn at EDT and say "you think you can change reality by managing the news!" The words "flagrantly, confidently, egregiously wrong" get bandied about, at least by one person. Everyone thinks everyone else is crazy. There is also a curious process by which an XDT'er, for any value of X, responds to counter...
Memory bandwidth constraints imply economies of scale in AI inference
by Ege Erdil — 2023-09-17T14:01:34.701Z

Contemporary GPUs often have very imbalanced memory vs arithmetic operation capabilities. For instance, an H100 can do around 3e15 8-bit FLOP/s, but the speed at which information can move between the cores and the GPU memory is only 3 TB/s. As 8 bits = 1 byte, there is a mismatch of three orders of magnitude between the arithmetic operation capabilities of the GPU and its memory bandwidth.

This imbalance ends up substantially lowering the utilization rate of ML hardware when batch sizes are small. For instance, suppose we have a model parametrized by 1.6 trillion 8-bit floating point numbers. To just fit the parameters of the model onto the GPUs, we'll need at least 20 H100s, as each H100 has a VRAM of 80 GB. Suppose we split our model into 20 layers and use 20-way tensor parallelism: this means that we slice the parameters of the model "vertically", such that the first GPU holds the first 5% of the parameters in every layer, the second GPU holds the second 5%, et cetera.

This sounds good, but now think of what happens when we try to run this model. In this case, roughly speaking, each parameter comes with one addition and one multiplication operation, so we do around 3.2 trillion arithmetic operations in one forward pass. As each H100 does 3e15 8-bit FLOP/s and we have 20 of them running tensor parallel, we can do this in a mere ~ 0.05 milliseconds. However, each parameter also has to be read into memory, and here our total memory bandwidth is only 60 TB/s, meaning for a model of size 1.6 TB we must spend (1.6 TB)/(60 TB/s) ~= 27 ms just because of the memory bottlenecks! This bottlenecks inference and we end up with an abysmal utilization rate of approximately (0.05 ms)/(27 ms) ~= 0.2%. This becomes even worse when we also take in inter-GPU communication costs into account, which would be at around 1 TB/s if the GPUs are using NVLink.

Well, this is not very good. Most of our arithmetic operation capability is being wasted because the ALUs spend most of their time idling and waiting for the parameters to be moved to the GPU cores. Can we somehow improve this?

A crucial observation is that if getting the parameters to the GPU cores is the bottleneck, we want to somehow amortize this over many calls to the model. For instance, imagine we could move a batch of parameters to the cores and use them a thousand times before moving on to the next batch. This would do...
How to talk about reasons why AGI might not be near?
by Kaj_Sotala — 2023-09-17T08:18:31.100Z

I occasionally have some thoughts about why AGI might not be as near as a lot of people seem to think, but I'm confused about how/whether to talk about them in public.

The biggest reason for not talking about them is that one person's "here is a list of capabilities that I think an AGI would need to have, that I don't see there being progress on" is another person's "here's a roadmap of AGI capabilities that we should do focused research on". Any articulation of missing capabilities that is clear enough to be convincing, seems also clear enough to get people thinking about how to achieve those capabilities.

At the same time, the community thinking that AGI is closer than it really is (if that's indeed the case) has numerous costs, including at least:

Immense mental health costs to a huge number of people who think that AGI is imminent
People at large making bad strategic decisions that end up having major costs, e.g. not putting any money in savings because they expect it to not matter soon
Alignment people specifically making bad strategic decisions that end up having major costs, e.g. focusing on alignment approaches that one might pay off in the long term and neglecting more foundational long-term research
Alignment people losing credibility and getting a reputation of crying wolf once predicted AGI advances fail to materialize

Having a better model of what exactly is missing could conceivably also make it easier to predict when AGI will actually be near. But I'm not sure to what extent this is actually the case, since the development of core AGI competencies feels more of a question of insight than grind[1], and insight seems very hard to predict.

A benefit from this that does seem more plausible would be if the analysis of capabilities gave us information that we could use to figure out what a good future landscape would look like. For example, suppose that we aren't likely to get AGI soon and that the capabilities we currently have will create a society that looks more like the one described in Comprehensive AI Services, and that such services could safely be used to detect signs of actually dangerous AGIs. If this was the case, then it would be important to know that we may want to accelerate the deployment of technologies that are taking in the world in a CAIS-like direction, and possibly e.g. promote rather than oppose things like open source LLMs.

One argum...
Contra Heighn Contra Me Contra Heighn Contra Me Contra Functional Decision Theory

by Bentham's Bulldog — 2023-09-17T02:15:41.430Z

Heighn and I have an ongoing debate about FDT. This is my reply to Heighn’s reply to my reply to Heighn’s reply to my post. You can trace back the links from

here

—which is the article I’m responding to. This will probably be my last post on the topic, so let me boil the dispute down to 4 things we disagree about. I will then argue I am right on each topic. This article might be sort of drab! There will be a lot of “I say X—Heighn’s reply is Y, my reply is Z, Heighn’s reply is A, my reply is B.” If you want to get a grasp of the core of the disagreement you’ll probably only need to read section 1—the others are just haggling over the details.

1 Is there a divergence between what’s rational to do and what type of agent you should want to be?

This may just be the crux of our disagreement. I claim there is no difference here: the questions What type of agent do I want to be? and What decision should I make in this scenario? are equivalent. If it is wise to do X in a given problem, then you want to be an X-ing agent, and if you should be an X-ing agent, then it is wise to do X. The only way to do X is to have a decision procedure that does X, which makes you an X-ing agent. And if you are an X-ing agent, you have a decision procedure that does X, so you do X.

This quote from Heighn I think expresses the core of the disagreement. Heighn claims that it’s rational to do X if and only if it’s best to be the type of agent who does X. Heighn asserts that if it’s wise to do X in a given problem, then you want to be an X-ing agent—seemingly without argument. So the question is, is this a plausible stipulation. Specifically, let’s think about a case I gave earlier that Heighn provides a very precise summary of:

Unfortunately, omnizoid once again doesn't clearly state the problem - but I assume he means that

there's an agent who can (almost) perfectly predict whether people will cut off their legs once they exist

this agent only creates people who he predicts will cut off their legs once they exist

existing with legs > existing without legs > not existing

Heighn’s diagnosis of such a case is that it’s rational to cut off your legs once you exist because the types of agents who do that are more likely to exist and thus do better. I think this is really implausible—once you exist, you don’t care about the probability of agents like you existing. You already exist! Why do...
Keeping Dresser Drawers In
by jefftk — 2023-09-17T02:10:03.458Z

We have an old dresser in our kitchen we use as extra cabinet space:

As an old dresser with wooden runners, however, there's nothing to
keep the drawers from being pulled all the way out, and falling with a
crash. Which is a problem, since we have an inquisitive toddler. There
are various approaches for making drawer stops, but to figure out what
might be a good fit for ours I started by taking the drawer all the way
out:

Many
designs attach to the back of the drawer, but ours seemed kind of
flimsy there and there was quite a bit of room on the sides, so I
decided I would just put a screw on each side:

It works well! The drawers move freely but can't come all the way out
anymore:

If at some point we need to remove the drawers to move the dresser
I can take the screws back out.