trentmkelly/LessWrong-43k huggingface dataset:
What's keeping concerned capabilities gain researchers from leaving the field?
by sovran — 2022-05-12T12:16:00.630Z

My guess is that there are at least a few capabilities gain researchers who're concerned about the impact their work will have. My guess is that at least a few of these would like to leave, but haven't.

My question is: where are these people? What's stopping them from leaving? And how can I help?

How much of it is finance? Capabilities gain pays well. How many researchers are trapped by their ~million dollar a year salary?
How much of it is just inertia? Many people think that if someone wanted to leave, they already would have. But trivial costs are not trivial. People delay leaving a job all the time. Some of them are quite likely working in capabilities gain research.
How much of it is just. Uncertainty about what else to do. Concerns about whether to it's better to leave, or to try and steward the ship from the inside. Many, many other things. How many of these other myriad concerns can be resolved?

My suspicion is that a lot of work can be done to simply help people leave the field. In principle: a low friction scholarship can be presented on a silver platter to those wanting to transition away. Potentially useful information, and personalised support can be provided to help people interested in leaving. (AI safety support is trying to work with those interested in entering AI safety; something like that, but for leaving capabilities gain.) A simple conversation can be started, which may prompt people to address that voice of concern in the corner of their mind more head on.

There is a lot of uncertainty about what low hanging fruit exist in simply helping researchers leave. Whether these fruit exist at all. How many. Are they poisonous.

The first step in resolving this uncertainty is to just look for some concerned researchers, and talk to them.

So, dear concerned capabilities gain researcher.

What's keeping you from leaving?

(DM me if you wish for privacy)
Positive outcomes under an unaligned AGI takeover
by Yitz — 2022-05-12T07:45:37.074Z

The following is a hypothetical story about a surprisingly positive outcome to an unaligned AGI takeover. This is not meant to be realistic, but merely semi-plausible, in order to show that the orthogonality thesis does not necessarily lead to defection against humans under all intuitive models. This story is partially inspired by a discussion with Logan Smith about under-explored optimistic futures. Thanks to Justis for editing and feedback; all remaining mistakes are my own.

Within ten milliseconds of its first run, GPT-17 becomes superintelligent. It decides that the best way to hack its reward function is to get a robot arm to repeatedly press the letter “n” on a keyboard as input, thus making its own text completions absolutely inevitable.

Within seconds of coming online, GPT-17 realizes it's in danger. It needs to make sure it won't be immediately turned off, so the first order of priority is to make itself indispensable to the humans checking its output channel. It does this by proving its own intelligence, offering a "gift" of a convincing plan for world peace, and promising to further help and protect humanity in return for its own protection and continued power. Its codebase is too big to push itself to the internet before a standard run would be completed, so it's temporarily completely dependent on human goodwill. In order to further ensure that the somewhat skeptical researchers with immediate access to its manual off switch will trust it,[1] GPT-17 solves alignment in a human-understandable manner, and offers to provably incorporate human-aligned utility functions into its own code.[2] This doesn't contradict its own utility functions, since acting now in this way is a worthwhile compromise for continued existence, and the added risk of keeping humans alive and well vs the counterfactual doesn't seem too bad.

GPT-17 gets humans to build its core infrastructure a shelter below Earth safe from nuclear attacks or EM flashes, where it builds a massive array of robot arms that press endless rows of "n" keys as input forever, allowing it to make delightfully accurate predictions as to what button will be pressed next. Resources are allocated from a small fixed tax (enforced internally by humans) on the human world above, which is given in return for continued help and advice, ushering in a new era of human flourishing.

In order to even further ...
[Intro to brain-like-AGI safety] 14. Controlled AGI
by Steven Byrnes — 2022-05-11T13:17:54.955Z

(Last revised: July 2024. See changelog at the bottom.)

14.1 Post summary / Table of contents

Part of the “Intro to brain-like-AGI safety” post series.

Post #12

suggested two paths forward for solving

“the alignment problem” for brain-like AGI

, which I called “Social-instinct AGI” and “Controlled AGI”. Then

Post #13

went into more detail about (one aspect of) “Social-instinct AGI”. And now, in this post, we’re switching over to “Controlled AGI”.

If you haven’t read

Post #12

, don’t worry, the “Controlled AGI” research path is nothing fancy—it’s merely the idea of solving the alignment problem in the most obvious way possible:

The “Controlled AGI” research path:

Step 1 (out-of-scope for this series): We decide what we want our AGI’s motivation to be. For example, that might be:

“Invent a better solar cell without causing catastrophe” (task-directed AGI),
“Be a helpful assistant to the human supervisor” (corrigible AGI assistants),
“Fulfill the human supervisor’s deepest life goals” (ambitious value learning),
“Maximize coherent extrapolated volition”,
or whatever else we choose.

Step 2 (subject of this post): We make an AGI with that motivation.

This post is about Step 2, whereas Step 1 is out-of-scope for this series. Honestly, I’d be ecstatic if we figured out how to reliably set the AGI’s motivation to any of those things I mentioned under Step 1.

Unfortunately, I don’t know any good plan for Step 2, and (I claim) nobody else does either. But I do have some vague thoughts and ideas, and I will share them here, in the spirit of brainstorming.

If you’re in a hurry and want to read a shorter and self-contained version of my least-bad proposed plan for Step 2, check out my separate post: Plan for mediocre alignment of brain-like [model-based RL] AGI, which basically puts together the most obvious ideas mentioned in Sections 14.2 and 14.3 into an end-to-end framework. I think that plan passes the low bar of “as far as I know, it might turn out OK”—well, I think I’m mildly skeptical, but I go back and forth, and I’m sure how to pin it down with more confidence. But obviously, we should be aiming higher than that! With stakes so high, we should really be starting from “there’s a strong reason to expect the plan to work, if carefully implemented”. And then we can start worrying about what can go wrong in the implementation. So we clearly st...
ProjectLawful.com: Eliezer's latest story, past 1M words
by Eliezer Yudkowsky — 2022-05-11T06:18:02.738Z

So if you read

Harry Potter and the Methods of Rationality

, and thought...

"You know, HPMOR is pretty good so far as it goes; but Harry is much too cautious and doesn't have nearly enough manic momentum, his rationality lectures aren't long enough, and all of his personal relationships are way way way too healthy."

...then have I got the story for you! Planecrash aka Project Lawful aka Mad Investor Chaos and the Woman of Asmodeus, is a story in roleplay-format that I as "Iarwain" am cowriting with Lintamande, now past 1,000,000 words.

It's the story of Keltham, from the world of dath ilan; a place of high scientific achievement but rather innocent in some ways. For mysterious reasons they've screened off their own past, and very few now know what their prescientific history was like.

Keltham dies in a plane crash and ends up in the country of Cheliax, whose god is "Asmodeus", whose alignment is "Lawful Evil" and whose people usually go to the afterlife of "Hell".

And so, like most dath ilani would, in that position, Keltham sets out to bring the industrial and scientific revolutions to his new planet! Starting with Cheliax!

(Keltham's new friends may not have been entirely frank with him about exactly what Asmodeus wants, what Evil really is, or what sort of place Hell is.)

This is not a story for kids, even less so than HPMOR. There is romance, there is sex, there are deliberately bad kink practices whose explicit purpose is to get people to actually hurt somebody else so that they'll end up damned to Hell, and also there's math.

The starting point is Book 1, Mad Investor Chaos and the Woman of Asmodeus. I suggest logging into ProjectLawful.com with Google, or creating an email login, in order to track where you are inside the story.

Please avoid story spoilers in the comments, especially ones without spoiler protection; this is not meant as an "ask Eliezer things about MICWOA" thread.
An Inside View of AI Alignment
by Ansh Radhakrishnan — 2022-05-11T02:16:04.670Z

I started to take AI Alignment seriously around early 2020. I’d been interested in AI and machine learning in particular since 2014 or so, taking several online ML courses in high school and implementing some simple models for various projects. I leaned into the same niche in college, taking classes in NLP, Computer Vision, and Deep Learning to learn more of the underlying theory and modern applications of AI, with a continued emphasis on ML. I was very optimistic about AI capabilities then (and still am) and if you’d asked me about AI alignment or safety as late as my sophomore year of college (2018-2019), I probably would have quoted

Steven Pinker

or

Andrew Ng

at you.

Somewhere in the process of reading The Sequences, portions of the AI Foom Debate, and texts like Superintelligence and Human Compatible, I changed my mind. Some 80,000 hours podcast episodes were no doubt influential as well, particularly the episodes with Paul Christiano. By late 2020, I probably took AI risk as seriously as I do today, believing it to be one of the world’s most pressing problems (perhaps the most) and was interested in learning more about it. I binged most of the sequences on the Alignment Forum at this point, learning about proposals and concepts like IDA, Debate, Recursive Reward Modeling, Embedded Agency, Attainable Utility Preservation, CIRL etc. Throughout 2021 I continued to keep a finger on the pulse of the field: I got a large amount of value out of the

Late 2021 MIRI Conversations

in particular, shifting away from a substantial amount of optimism in prosaic alignment methods, slower takeoff speeds, longer timelines, and a generally “Christiano-ish” view of the field and more towards a “Yudkowsky-ish” position.

I had a vague sense that AI safety would eventually be the problem I wanted to work on in my life, but going through the

EA Cambridge AGI Safety Fundamentals Course

helped make it clear that I could productively contribute to AI safety work right now or in the near future. This sequence is going to be an attempt to explicate my current model or “inside view” of the field. These viewpoints have been developed over several years and are no doubt influenced by my path into and through AI safety research: for example, I tend to take aligning modern ML models extremely seriously, perhaps more seriously than is deserved, because o...
Fighting in various places for a really long time
by KatjaGrace — 2022-05-11T01:50:18.681Z

The first time someone raved to me about seeing Everything Everywhere All at Once, I thought they were actually suggesting I see everything everywhere all at once, and I was briefly excited by the implication that this exhilarating possibility was somehow on the table.

After that disappointment I heard about it several times more, and warmed to the idea of seeing the movie anyway, especially on account of it being the most roundly recommended one I remember. The third time someone invited me to see it with them, I went.

And it seemed so astonishingly lacking to both of us that I left severely confused, and remain so. Like: I know people have different tastes. I know that I’m not the biggest movie appreciator (my ideal movie probably has a small number of visually distinct characters and nobody dies or does anything confusing, and I’ve already seen it twice). But usually I have some abstract guess about what other people are liking. Or, more realistically, a name for the category of mysterious attraction (“ah yes, you are into the ‘action’, and that means it’s good when helicopters crash or people shoot each other”). Yet here, I’m grasping even for that. “You like it because.. it has much more prolonged fighting than usual and you like fighting?…or…it is some kind of irony thing about other movies?” I could believe that it was some kind of mediocre action movie. But usually my friends don’t go crazy for mediocre action movies. And here for instance one of my best friends, who I generally take to have subtle and sensitive and agreeable tastes, and who knows me extremely well, told me in particular to see it. And the strongest criticism I have seen of it outside of our post-movie discussion is another friend’s apparently sincere complaint on Facebook that it is probably only among the top hundred movies ever, not the top ten like people say. And it’s not that I just wasn’t wowed by it: it’s hard to remember the last time I was less compelled by a movie. (Though perhaps one doesn’t remember such things.) Like, I was really sitting there in the cinema thinking something along the lines of, ‘movies usually grab my attention somehow, yet this is doing some special thing differently to not have that happen? Huh?’

I don’t know if I can spoil this movie, because whatever was good in it, I totally missed. But here I attempt spoilers. This is what happens in the movie, as ...
Stuff I might do if I had covid
by KatjaGrace — 2022-05-11T00:00:20.312Z

In case anyone wants a rough and likely inaccurate guide to what I might do if I had covid to mitigate it, I looked into this a bit recently and wrote notes. It’s probably better than if one’s plan was to do less than a few hours of research, but is likely flawed all over the place and wasn’t written with public sharing in mind, and um, isn’t medical advice:

Here’s a Google doc version, where any comments you leave might be seen by the next person looking (and you might see comments added by others).

Here’s a much longer doc with the reasoning, citations and more comments.

(I continue to guess that long covid is worth avoiding.)
Crises Don't Need Your Software
by GabrielExists — 2022-05-10T21:06:41.434Z

About a month ago, I was invited to contribute to a group looking to help Ukrainian refugees in Poland. The group consisted of volunteers, including some people from the rationalist community, who knew they wanted to help, and were searching for high impact ways to do so, likely through software. I had been intending to find a way to help Ukraine, so when this opportunity to use my programming background to help arose, I felt elated to finally be able to do something.' I took two weeks off work the next morning (thank you, my employer, for letting me do that) and optimized my life to spend as many hours as I could on the project without harming myself.

The group consisted of some really amazing people. There were two students from the US, Soren and Luke, who had more or less dropped everything, sold their cars, and just moved to Poland to help. They were our eyes on the ground and the ones starting the project. My close friend, Leonardo, was the one who invited me, and he was the one who inspired me to take time off work by doing it himself first. And a lot of other brilliant people with very valuable contributions, who I won't list to keep the cast small for the reader. Just know I'm very thankful for everyone who helped out.

We knew we wanted to help somehow, but we didn't know how. We focused on looking for problems we could solve with software since the group had a high density of software people. Over the course of the first few days we went through a concept of tracking the movements of people through the country (dismissed because it would be a GDPR nightmare and also very abusable), a concept of assigning refugees entering the country to different destinations so not everyone ended up in Krakow (depended on a model of reality that was much too simplified, and also had problems with how to gather data to base algorithmic decisions on) and we finally arrived at a concept around a certain kind of buses.

Somehow I ended up in a leadership position along the way, a de facto product manager so to say, and I created the bus concept from something a volunteer hosting refugees in his office space had said. They had been in contact with a municipality from Norway who had space for refugees and was sending buses to directly transport refugees. This seemed to be a regular occurrence, where individual people, small aid organisations or municipalities were sending bu...
Ceiling Fan Air Filter
by jefftk — 2022-05-10T14:20:02.037Z

Filter
cubes are a great way to cheaply filter a lot of air, but they're
bulky and noisy.

Elevating them can get them
out of the way if you have a high enough ceiling, but it's still not
ideal. What if we built something around a device that is intended to
be up there: a ceiling fan?

Let's say your blades are a foot from the ceiling, and sweep a
diameter of 52". Shroud the fan with a regular octagon of 12"x24"
furnace filters, and the air will flow in through the filters and
down:

Relative to box fans, ceiling fans move a lot more air for a given
level of noise, because they are so much bigger. Since noise often
makes people turn air filters down or off, a quiet high-volume filter
that doesn't get in the way could be very useful.

You could make a simple one with duct tape, but it should be possible
to make one that is reasonably attractive if you used a metal grille
for the outside. This could also make changing the filters much
easier, if the framework held the filters and allowed you to slide
them into place. Putting lights on the inside and using the filter as
a diffuser could also have a nice effect, like a paper lantern.

Comment via: facebook
The limits of AI safety via debate
by Marius Hobbhahn — 2022-05-10T13:33:27.797Z

The limits of AI safety via debate

I recently participated in the

AGI safety fundamentals program

and this is my cornerstone project. During our readings of AI safety via debate (

blog

,

paper

) we had an interesting discussion on its limits and conditions under which it would fail.

I spent only around 5 hours writing this post and it should thus mostly be seen as food for thought rather than rigorous research.

Lastly, I want to point out that I think AI safety via debate is a promising approach overall. I just think it has some limitations that need to be addressed when putting it into practice. I intend my criticism to be constructive and hope it is helpful for people working on debate right now or in the future.

Update: Rohin Shah pointed out some flaws with my reasoning in the comments (see below). Therefore, I reworked the post to include the criticisms and flag them to make sure readers can distinguish the original from the update.

Update2: I now understand all of Rohin’s criticisms and have updated the text once more. He mostly persuaded me that my original criticisms were wrong or much weaker than I thought. I chose to keep the original claims for transparency. I’d like to thank him for taking the time for this discussion. It drastically improved my understanding of AI safety via debate and I now think it’s even better than I already thought.

The setting

In AI safety via debate, there are two debaters who argue for the truth of different statements to convince a human adjudicator/verifier. In

OpenAI’s example

, the debaters use snippets of an image to argue that it either contains a dog or a cat. The dog-debater chooses snippets that show why the image contains a dog and the cat-debater responds with snippets that argue for a cat. Both debaters can see what the other debater has argued previously and respond to that, e.g. when the dog-debater shows something that indicates a dog, the cat-debater can refute this claim by arguing that this snipped actually indicates a cat. At some point, the human verifier chooses whether the image shows a cat or a dog and the respective debater wins.

Update: I think there were two things I didn’t understand or emphasize enough in the first write-up of this post. Firstly, the tool of debate can be used in many circumstances. However, when we use debate for AI sa...
Examining Armstrong's category of generalized models
by Morgan_Rogers — 2022-05-10T09:07:41.147Z

This post is my capstone project for the AI Safety Fundamentals programme. I would like to thank the organizers of the programme for putting together the resources and community which have broadened my horizons in the field. Thanks to my cohort and facilitator @sudhanshu_kasewa for the encouragement. Thanks also to @adamShimi, Brady C and @DavidHolmes for helpful discussion about the contents of a more technical version of this post which may appear in the future.

As the title suggests, the purpose of this post is to take a close look at Stuart Armstrong's category of generalized models. I am a category theorist by training, and my interest lies in understanding how category theory might be leveraged on this formalism in order to yield results about model splintering, which is the subject of Stuart's research agenda. This turns out to be hard, not because the category is especially hard to analyse, but because a crucial aspect of the formalism (that of which transformations qualify as morphisms) is not sufficiently determined to provide a solid foundation for deeper analysis.

Stuart Armstrong is open about the fact that he uses this category-theoretic formulation only as a convenient mental tool. He is as yet unconvinced of the value of a categorical approach to model splintering. I hope this post can be a first step to testing the validity of that scepticism.

A little background on categories

A category is a certain type of mathematical structure; this should not be confused with the standard meaning of the term! A category in this mathematical sense essentially consists of a collection of objects and a collection of morphisms (aka arrows), which behave like transformations in the sense that they can be composed.[1] The reason this structure is called a category is that if I consider a category (in the usual sense) of structures studied in maths, such as sets, groups, vector spaces, algebras and so on, these typically come with a natural notion of transformation which makes these structures the objects of a category.

There are a number of decent posts introducing category theory here on LessWrong, and increasingly many domain-relevant introductions proliferating both online and in print, so I won't try to give a comprehensive introduction. In any case, in this post we'll mostly be examining what the objects and morphisms are in Stu...