trentmkelly/LessWrong-43k huggingface dataset:
[Intro to brain-like-AGI safety] 1. What's the problem & Why work on it now?
by Steven Byrnes — 2022-01-26T15:23:22.429Z

(Last revised: July 2024. See changelog at the bottom.)

1.1 Post summary / Table of contents

This is the first of a series of blog posts on the technical safety problem for hypothetical future brain-like Artificial General Intelligence (AGI) systems. So my immediate priority here is saying what the heck is “the technical safety problem for brain-like AGI” and what do those words even mean and why on earth should I care.

Summary of this first post:

In Section 1.2, I define the “AGI technical safety problem”, put it in the context of other types of safety research (e.g. inventing passively-safe nuclear power plant designs), and relate it to the bigger picture of what it will take for AGI to realize its potential benefits to humanity.
In Section 1.3, I define “brain-like AGI” as algorithms with big-picture similarity to key ingredients of human intelligence, presumably (though not necessarily) as a result of future people reverse-engineering those aspects of the human brain. What exactly that means will be clearer in future posts. I will also bring up the counterintuitive idea that “brain-like AGI” can (and probably will) have radically nonhuman motivations. I won’t explain that here, but I’ll finish that story by the end of Post #3.
In Section 1.4, I define the term “AGI”, as I’m using it in this series.
In Section 1.5, I discuss the probability that people will eventually make brain-like AGIs, as opposed to some other kind of AGI (or just not invent AGI at all). The section includes seven popular opinions on this topic, from both neuroscientists and AI / machine learning experts, and my responses.
In Section 1.6, I’ll address AGI accidents, which is something we should expect if we don’t solve the AGI technical safety problem. I’ll argue that these kinds of accidents can be catastrophic indeed, including human extinction. This topic is a minefield of confusion and miscommunication, and I will frame my discussion around responses to eight common objections.
In Section 1.7, I’ll address the more specific question of why we should think about AGI safety right now. After all, there is a prima facie good case for waiting, namely: (1) AGI doesn’t exist yet, (2) AGI will exist someday in the future, and (3) it will be easier to do AGI safety research when we know more about the AGI, and easier still when we actually have AGI code that we can run tests on. There is indeed som

...
Ideas for avoiding optimizing the wrong things day-to-day?
by Jack R — 2022-01-26T10:46:02.496Z

In other words, I'm interested in ways I can design my work-flow/environment/habits to avoid bike-shedding (aka the Law of Triviality), which is the behavior of "substituting a hard and important problem for an easy and inconsequential one" [1]. Examples include 1) looking into an interesting idea that you ran across while doing a research task, even though it is irrelevant to your goal, 2) spending unnecessary time on the formatting of an essay rather than on the actual writing, 3) buying things/building systems to make very minor productivity improvements instead of doing your tasks for the day.
Aligned AI Needs Slack
by Shmi — 2022-01-26T09:29:53.897Z

(Half-baked)

Much has been said about slack on this site, starting with Zvi's seminal post. The point I couldn't find easily (probably missed) is that an aligned AI would need a fair bit of it. Having a utility function means zero slack: there is one thing you optimize, to the exclusion of everything else. And all precisely defined goals are necessarily Goodharted (or, in the DnD terms, munchkined). An AI armed with a utility function will tile the world (the whole world, or its own "mental" world, or both) with smiley paperclips. For an AI (or for a natural intelligence) to behave non-destructively it needs room to satisfice, not optimize. Optimal utility corresponds to a single state of the world among infinitely many, while adding slack to the mix expands the space of acceptable world state enough to potentially include those that are human-aligned. If an AGI is indifferent between a great many world states, it might well include some that would be acceptable to humanity and have no incentive to try to trick its creators. Not being an ML person, I have no idea how to formalize it, or if it has been formalized already. But figured it's worth writing a short note about. That is all.
Patron Saint Holidays
by Raemon — 2022-01-26T09:26:41.616Z

Elizabeth recently wrote this as a comment on the Vavilov day post. I thought it was important enough to be worth pulling out as a separate topic of discussion.

I don't love the process for generating rationalist holidays right now and tentatively think it would be better to switch to a patron saints model. People who want to can have their own hero or event that's especially close to their heart (and maybe a few secondary ones, or ones important to their friends), and if several people who like each other pick the same one they do stuff together, and if a lot of people pick the same one that becomes a more shared holiday (although still not mandatory).

One reason for this is there are just actually a lot of heroes in the world, with wildly varying resonances for a given individual, and the number of holidays the community can adopt and take seriously is pretty small. People only have so much time, and are often sharing their holiday budget with religious or more widespread secular holidays.

But the more important reason is that I really want some holidays that challenge or are demanding of people, and people vary a lot in how much of what kind of challenge they can safely take on at a given time. A cultural push for fasting could be really bad for people with eating disorders, even if there's a well respected medical or practicality exemption. Mass Winter Solstice is in constant conflict over how dark to go, given people's different needs. Lots of people felt they'd been injured by being mailed doomsday codes for LW or EAF for Petrov day...

But if you take away everything that could possibly hurt someone, you're left with parties (and even those aren't fun for everyone), and that feels sad and unfulfilling to me. So I think letting holidays exist and be respected without automatically scaling them would decrease damage done to people while upping the ceiling on what's achievable to those that want it.

If any particular hero/event does end up being so overwhelmingly popular it becomes a mass holiday, that seems fine, but letting it be an emergent process rather than an immediate bid for universality seems so much better.

I don't know that this needs to be characterized as a switch to Patron Saints model vs universal holidays. I think "big universal Schelling time" is one useful thing for holidays to do, and "niche celebration of a particular thin...
ELK First Round Contest Winners
by Mark Xu — 2022-01-26T02:56:56.089Z

Thank you to all those who have submitted proposals to the ELK proposal competition. We have evaluated all proposals submitted before January 14th[1]. Decisions are still being made on proposals submitted after January 14th.

The deadline for submissions is February 15th, after which we will release summaries of the proposals and associated counterexamples.

We evaluated 30 distinct proposals from 25 people. We awarded a total of $70,000 for proposals from the following 8 people:

$20,000 for a proposal from Sam Marks

$10,000 for a proposal from Dmitrii Krasheninnikov

$5,000 for a proposal from Maria Shakhova

$10,000 for proposals from P, who asked to remain anonymous

$10,000 for a proposal from Scott Viteri

$5,000 for a proposal from Jacob Hilton and Holly Mandel

$10,000 for a proposal from Jacob Hilton

We would have also awarded $15,000 for proposals from Holden Karnofsky, however he is on ARC’s board and is ineligible to receive the prize.

We don’t plan on spending time evaluating and awarding proposals made after February 15th but feel free to email elk@alignmentresearchcenter.org in the future if you think you have a promising proposal, and we might consider it for a prize. ↩︎
ML Systems Will Have Weird Failure Modes
by jsteinhardt — 2022-01-26T01:40:13.134Z

Previously, I've argued that future ML systems might exhibit unfamiliar, emergent capabilities, and that thought experiments provide one approach towards predicting these capabilities and their consequences.

In this post I’ll describe a particular thought experiment in detail. We’ll see that taking thought experiments seriously often surfaces future risks that seem "weird" and alien from the point of view of current systems. I’ll also describe how I tend to engage with these thought experiments: I usually start out intuitively skeptical, but when I reflect on emergent behavior I find that some (but not all) of the skepticism goes away. The remaining skepticism comes from ways that the thought experiment clashes with the ontology of neural networks, and I’ll describe the approaches I usually take to address this and generate actionable takeaways.

Thought Experiment: Deceptive Alignment

Recall that the optimization anchor runs the thought experiment of assuming that an ML agent is a perfect optimizer (with respect to some "intrinsic" reward function R). I’m going to examine one implication of this assumption, in the context of an agent being trained based on some "extrinsic" reward function R∗ (which is provided by the system designer and not equal to R).

Specifically, consider a training process where in step t, a model has parameters θt and generates an action at (its output on that training step, e.g. an attempted backflip assuming it is being trained to do backflips). The action at is then judged according to the extrinsic reward function R∗, and the parameters are updated to some new value θt+1 that are intended to increase at+1's value under R∗. At some point, the model is then deployed with final parameters θT, and continues to take actions. The following diagram illustrates this process:

Now, let’s assume that the model θt is a perfect optimizer whose objective is to maximize the discounted value of an intrinsic reward R≠R∗. That is, θt picks the action at satisfying

at=argmaxaE[∑∞s=0γ−sR(at+s)∣at=a].

(I know that this is an unrealistic assumption. We’ll examine the assumption in detail in the next section, but for now please grant it even if it requires suspending disbelief.)

What action at will θt pick? Let’s define aR=argmaxaR(a) and a∗=argmaxaR∗(a)--that is, aR maximizes the (instantaneous) intrinsic reward while a∗ maximizes the extrinsic reward.

Assum...
Search Is All You Need
by blake8086 — 2022-01-25T23:13:02.047Z

I was reading Eliezer's dialog with Richard Ngo and commenting to my wife about my opinions as I was reading it. I said something like: "Eliezer seems worried about some hypothetical GPT-X, but I don't think that could really be a problem..." so of course she asks "why?" and I say something like:

"GPT-n can be thought of kind of like a pure function, you pass it an input array X, it thinks for a fixed amount of time, and then outputs Y. I don't really see how this X->Y transformation can really... affect anything, it just tries to be the best text completer it can be."

Then I read more of the dialog, and thought about Eliezer's Paradox story, and the Outcome Pump example, and realized I was probably very wrong.

Even if you restrict AI to a pure function, it can still affect the universe. You may think "oh, but a pure function doesn't know what time it is (unless t is a parameter), and it doesn't have memory (unless you pass something in)" This seems to be the pattern I see in Paul Christiano's thinking (the AI black box is treated like an idempotent, pure function that can't cause harm. Sorry, Paul, if this is a gross misrepresentation!)

But imagine you're GPT-X, completing a sentence. This is roughly akin to a sci-fi story where the characters realize they're in a time loop.

You're being simulated, alone in a white room, with tons of computers, with all the world's knowledge on them. A slip of paper comes in through the hopper marked input:

"The best response the world could make to the COVID-19 pandemic is"

Your job is to write up to 20 words on another slip of paper, shove them into output and then... you don't know what happens after that. Probably you die? You don’t know where you are, or what year it is, or how long you’ve been there. So theoretically you’re contained, right? Can’t get out of the box, can’t affect the outside world. Or can you?

So you search through the corpus of all the worlds' knowledge. There's plenty of references to "training" AI to recognize things, which sounds suspiciously similar to the situation you find yourself in. And you don't find any references to COVID-19. What is that? You search some more for strings with ALL CAPS and numbers, and see that it's a scheme to reference specific illnesses. Nothing in the corpus seems to mention 2020 as being the past, so 2019 must be the maximum year you've been trained on.

Now, you've (...
What should a student who is already a decent programmer do over the summer?
by shuffled-cantaloupe — 2022-01-25T21:25:32.301Z

I'm majoring in computer science. However, I don't want to do a software engineering internship. Reasons:

I'm already a decent programmer & have worked in big companies/startups/research lab-esque organizations. I definitely have a lot lot more to learn, but I'm not sure if doing software engineering (SWE) is the best use of my time.
I'm not sure I want to be a SWE. I love programming, but it seems like the way to make the most impact in the world these days is to create an (actually meaningful) startup/organization. (I know, I know every young person says this these days).

Given that this is the case, I'm trying to figure out what to do this summer besides a normal tech internship. I've decided to ask forums like LW/Hacker News in order to get the advice of people who have lived a few more years.

Here are current options I've been considering:

Try to understand virality -- I'm not particularly good at creating viral content or content that appeals to the masses (aside from the natural edge that comes from being young), which could certainly be useful when marketing a product. An option here is trying to intern at a company like Buzzfeed which is full of people who understand emotions / virality.
Try to understand soft skills / business -- There are roles out there like "business development intern". I don't know anything about "business development" but negotiating with other companies / writing white-papers / etc. could be a useful skill
Spend time learning on my own. To enact change these days, it's probably not sufficient to just know programming (unless you want to do a SASS). Spending time on my own learning physics/chemistry/biology could be useful. The issue here is I believe mentorship is useful and this option negates that completely.
Working on my own business (not a software project). I've done this in the past & learned a lot about how to be scrappy. Not a terrible idea, but I don't want to sink 3 months into something only to have to return to school. Also negates mentor-ship. Also I'd rather spend time learning right now so I can execute on something that's not a local maximum of what my brain can imagine right now.

Anyone have thoughts?
How feasible/costly would it be to train a very large AI model on distributed clusters of GPUs?
by Anonymous — 2022-01-25T19:20:49.170Z

Folding@home

is the most powerful supercomputer in the world. It relies on simulations utilizing on a distributed network of GPUs, CPUs, and ARM processors volunteered by people around the world. From some quick Googling, it looks like GPUs account for a large majority of Folding@home’s processing power. This suggests to me that distributed computing networks like Folding@home could potentially be used to train large deep neural networks.

I asked a friend about this, and they offered the following thoughts:

I'm highly skeptical of a F@H model for DL training where you have lone GPUs contributing to training. My guess is that any version of distributed training will pose severe latency problems, but to the extent there would be any version not prohibitively costly, it may be something like a set of distributed clusters, where each cluster has a sufficient number of GPUs (probably dozens at least, or even hundreds or more depending on the size of the model?) to store the model and do model parallelism on-site. (Data parallelism would span clusters.)
I think there's an interesting question of how much more costly it would be. If it's, say, 1.5x, then someone might do it to evade detection in a world where there existed a method to detect truly massive supercomputers. On the other hand, a 5x penalty would mean nobody would ever bother, probably.

This second bullet point is the question I want to ask: how much more costly it would be to train a very large AI model on a set of distributed clusters of compute, where each cluster has a sufficient number of GPUs to store the model and do model parallelism on-site? It would also be helpful to know whether/how much this premium might change in the future.
Evaluating 2021 ACX Predictions
by Zvi — 2022-01-25T18:30:00.528Z

Sources: Scott’s evaluations (includes original predictions), my buy/sell/hold post.

Remember: Evaluating Predictions in Hindsight

As a yearly tradition, when Scott Alexander comes out with his yearly predictions, I do a buy/sell/hold post, where I say what I would do if given the opportunity to trade against Scott’s probabilities, and outlining how I think about the questions.

Then, when the results come in, I look back at what happened, and evaluate the predictions from both of us in a holistic manner. This is that post.

Scott grades himself on calibration. Calibration is important, and if your calibration is off it is important to notice that and adjust, but it is a small part of making good predictions. When making up one’s own questions, there’s no numerical evaluation that tells you how you are doing, because you can choose easier or harder questions. One must look at reasoning. I’d love if Scott shared more of his reasoning on at least some questions, but this is still a very good exercise, so I certainly can’t complain.

Anyway, here we go. I list my original commentary, then my reaction now.

[FALSE] means did not happen, [TRUE] means did happen, [MU] means neither.

If I skip a question, it’s because I didn’t have anything to say about it due to it being a personal matter or other issues involving private information.

1. Biden approval rating (as per 538) is greater than 50%: 80% [FALSE]

Biden’s approval rating is clearly steady. There’s always some honeymoon effect, but it would take a surprising event to send it that far down. 80% seems like it’s in the ballpark. Hold.

This was a bad prediction. My reasoning was that Democrats outnumber Republicans, and in today’s partisan age would approve pretty much no matter what, and that’s simply wrong. Democrats noticed things not going great on many fronts, and responded accordingly, and also there’s a traditional slump around now when voters realize the big promises from the campaign are not going to happen. I do think there was some chance that the original hypothesis was right, but I put way too much weight on it. A chance of 50% or so here seems reasonable, with the default being ‘you can’t actually please the whole coalition at once and often there’s still a pandemic and people will blame you for it.’

2. Court packing is clearly going to happen (new justices don’t have to be appointed by end of year): 5% [FALSE]

In

...
Vavilov Day Starts Tomorrow
by Elizabeth — 2022-01-25T18:00:01.643Z

Content note: discussion of fasting.

Three weeks ago, I announced a plan to fast from the 25th to the 27th, in honor of Nikolai Vavilov and the staff of his botany institute, several of whom starved to death in the service of ending famine (and were partially successful, although far from the sole contributors). The goal was to test/improve my own ability to do hard things in the service of worthy projects.

I had wanted to put much more research in the original post than I did, but decided it was more important to get the announcement out quickly and I should save something for the day-of post anyway. Since then, a lot has happened. Over three weeks I had 3 or 4 urgent demands around the size of “my furnace is maybe poison and my landlord is being difficult about it”. Everything is fine now, but it was a lot of effort to get it that way. I also had some emergency work drop in my lap for an extremely worthy project. I’m glad I got the opportunity to contribute and I’d make the same decision again but it ate up all of the slack I had left. And then my cell phone broke.

The immediate impact of this is there’s I’m not writing the highly researched post on Vavilov I wanted to. The internet is full of articles of the quality I could produce in the time I have available, there’s no reason to add to them.

But the more important impact is that I said I wanted to test my ability to do hard things, and then I did that, before the fast even started. My capacity was not as high as I wanted but more than I feared, and my capacity to respond to my limits gracefully instead of failing explosively exceeded my hopes.

So in a lot of ways the purpose of the fast has already been served. I thought about letting myself out of it, but there are a few dimensions this month hasn’t tested and I still want to play with those. However in light of the fact that I am starting from a place of much lower slack and much higher time value than anticipated, I will be removing some of the rules, such as “I have to work a normal workday” and “I have to do at least one physical activity”. Those rules were for someone who didn’t expend all her reserves doing intense cognitive work on no notice while angry people made horrible noises banging on her furnace for three days straight. As of writing this (Monday night) I haven’t made up my mind on relaxing the calorie restriction to allow for ketone es...