trentmkelly/LessWrong-43k huggingface dataset:
Shutting down AI is not enough. We need to destroy all technology.
by Matthew Barnett — 2023-04-01T21:03:24.448Z

A TIME article published recently calls for an “indefinite and worldwide” moratorium on new large AI training runs.

This moratorium would be better than no moratorium. I have respect for the author who wrote it. It’s an improvement on the margin.

I refrained from endorsing the essay because I think it is understating the seriousness of the situation and asking for too little to solve it.

If there was a plan for Earth to survive, if only we passed an indefinite and worldwide moratorium on large training runs, I would back that plan. There isn’t any such plan.

Here’s what would actually need to be done:

All human technology needs to be destroyed. There can be no exceptions, including for sharpened stones and hand axes. After everything is burned, we must then forget how to create fire. If a single exception is made, that increases the probability that civilization will be recreated within the next millennia and new large AI training runs will be started. If I had infinite freedom to write laws, I might carve out a single exception for technologies that prevent human diseases, like knowledge of germ theory; but if that was remotely complicating the issue I would immediately jettison that proposal and say to just shut it all down.

Shut down all the roads, melt all the cars. Burn down all of the tables and all of the books. Put a ceiling into how many calories of food any single human can furnish per day, and move it downward over the coming generations to compensate for the possibility that natural selection will keep making humans smarter. No exceptions for cloth or fireplaces. Destroy all human objects now to prevent them from moving to another country. Track all gazelles that are hunt. If anyone notices that a tribe is catching more gazelles than it should, be willing to choke them (with your bare hands, of course) one by one.

Shut it all down. Eliminate all technology. Dismantle modern civilization. Return to our primeval state.

We are not ready. We are not on track to be significantly readier any time in the next million years. If we go ahead on this, everyone will suffer, including children who did not choose this and did not do anything wrong.

Shut it down.
A policy guaranteed to increase AI timelines
by Richard Korzekwa  — 2023-04-01T20:50:14.832Z

The number of years until the creation of powerful AI is a major input to our thinking about risk from AI and which approaches are most promising for mitigating that risk. While there are downsides to transformative AI arriving many years from now, rather than few years from now, most people seem to agree that it is safer for AI to arrive in 2060 than in 2030. Given this, there is a lot of discussion about what we can do to increase the number of years until we see such powerful systems. While existing proposals have their merits, none of them can ensure that AI will arrive later than 2030, much less 2060.

There is a policy that is guaranteed to increase the number of years between now and the arrival of transformative AI. The General Conference on Weights and Measures defines one second to be 9,192,631,770 cycles of the optical radiation emitted during a hyperfine transition in the ground state of a cesium 133 atom. Redefining the second to instead be 919,263,177 cycles of this radiation will increase the number of years between now and transformative AI by a factor of ten. The reason this policy works is the same reason that defining a time standard works–the microscopic behavior of atoms and photons is ultimately governed by the same physical laws as everything else, including computers, AI labs, and financial markets, and those laws are unaffected by our time standards. Thus fewer cycles of cesium radiation per year implies proportionately fewer other things happening per year.

Making such a change might not sound politically tractable, but there is already precedent for making radical changes to the definition of a second. Previously it was defined in terms of Earth’s solar orbit, and before that in terms of Earth’s rotation. These physical processes and their implementations as time standards bear little resemblance to the present-day quantum mechanical standard. In contrast, a change that preserves nearly the entire standard, including all significant figures in the relevant numerical definition, is straightforward.

One possible objection to this policy is that our time standards are not entirely causally disconnected from the rest of the world. For example, redefining the time standard might create a sense of urgency among AI labs and the people investing in them. It’s not hard to imagine that the leaders and researchers within companies advancing the s...
Why I Think the Current Trajectory of AI Research has Low P(doom) - LLMs
by GaPa — 2023-04-01T20:35:18.248Z

[This is mostly posted as a some thoughts I wanted to check. I apologize that its messy and not complete. I needed to get something out there.]

This post explains the reasons why I think the probability of AGI killing everyone in the next few decades is very low, at least compared to what Yudkowsky argues.

1.1 By Far, Most Progress Toward AGI comes from LLMs

LLMs have offered such a huge boost towards AGI because they bootstrap the ability to reason by mimicing human reasoning.

Achieving this level of intelligence or even general knowledge through RL seems hardly more plausible now than at the inception of RL. Impressive progress has been made (Deepmind's AdA learning new RL tasks within the XLand 2.0 environment at human timescales) but this is not going to reproduce even BERT-level general knowledge anytime soon - which is essential for the ability to make impactful decisions in the real world.

I think we can also look at self driving to show this problem - even with huge compute budgets and huge dataset and competition in the space, self driving is still not solved. I think this field shows the progress of AI in general since it has high incentives to take advantage of every AI advance - I suspect it has received much more funding than LLMs and between real and simulated datasets, I would imagine the amount of data collected is comparable to internet text. Self-driving seems very simple compared to general intelligence, and yet somehow LLMs are probably(?) more able describe how to act on the road in a wide range of situations that a self driving car. e.g. that example of driving behind a truck transporting stoplights, or using theory-of-mind-adjacent-abilities to anticipate other drivers' actions.

I would argue that the closest progress to AGI outside LLMs is AI art generation, since text is a subset of images (you could imagine training diffusion on images of LLM datasets and doing in-painting for next-token-prediction). Text-image diffusion models already have general knowledge about physics, lighting, some amount of language, etc.. We've even seen that training diffusion models on audio spectrograms has produced SOTA music/voice generation models. However, this is all also based on learning from internet text/images using a denoising loss, so it has a very similar cap to LLMs, since both text and images cover similar ground (text can describe images)....
Repairing the Effort Asymmetry
by Duncan Sabien (Inactive) — 2023-04-01T20:23:34.261Z

Sometimes, you will say a thing!

You: "Proposition A."

A seems to you to be obviously true. It accords with your experience, doesn't violate any of the rules and laws of reality, makes sense with the rest of your model of the world. You'd like to move on from A to the actual more interesting conversation you want to have; A is just some necessary groundwork.

Sometimes, someone else will be DEEPLY SKEPTICAL of A. After all, different people have very different experiences of the world! Bubbles exist, and cultures are not universal.

Or perhaps it's not that they're deeply skeptical of A so much as that their brain reflexively transformed A into a B of which they are deeply skeptical (and they don't even notice that their B is not the same as your A).

Them: "What? How do you know that's true? Can you cite several specific examples? The best example I could come up with was [X], and since [X] is clearly ridiculous, your point is invalid!"

And sometimes, you sit there, absolutely confident that you could, in fact, spend 20 hours and 10,000 words to clear up all of the misunderstandings, and lay out all of the arguments in painstaking slow detail.

But you don't want to do that! You weren't trying to convince Every Rando of the truth of A, and you don't much care if This Rando doesn't get it, or runs off into the woods with their strawman.

But unfortunately, their misinterpretations can anchor others and skew the conversation, and a dangling unanswered "Cite specific examples?" comment accrues upvotes pretty quickly, and generates oft-undeserved skepticism through sheer representativeness. Surely if you had specific examples, you'd give them! Since you didn't give them, you must not have them!

(This, of course, ignores the fact that engagement is costly and effortful. Laying out thoughts takes time. Painstakingly correcting subtle misunderstandings is the work of hours or days or even weeks, involving a lot of getting into the weeds.)

And you didn't want to get into the weeds. You just wanted to make A explicit, so you could move on to the actually interesting conversation that takes place one level higher. You can see the ways that they subtly went wrong, and could, if you wanted, gently deconfuse them, one thought at a time, but you don't want to have to do that, as a prerequisite for having any interesting conversations at all.

It's one thing if you're feeling generous, an...
Draft: Inferring minimizers
by Alex_Altair — 2023-04-01T20:20:48.676Z

This post is a draft, a work in progress. Much of it will not make sense, and will not be optimized for the reader's experience. It does not necessarily reflect my current best understanding of optimization. Like the introduction draft, I'm opening it up because I've been working on this project for too long, and the world is moving too fast. I want people to more easily be able to interact with my research thoughts as I'm having them.

We left off with an argument by analogy of how we could use Kolmogorov complexity as an "objective" means to infer the presence of optimization in a system. But how exactly, does that analogy carry over? How exactly do we use K-complexity in the context of dynamical systems trajectories?

In this post, I'll treat what I would call the "easy" case. The cases are formed by the structure of the state orders, so let's dig deeper into that.

Order shapes

In the introduction post, I talked about how the optimization target or criterion could be represented by a mathematical object called an order. Orders get pretty interesting once you start to include orders over infinite sets. In some sense, you could say that finite orders differ only in how they order things, but that infinite orders can differ in how they are "shaped". Consider this example

The typical ordering over the integers
The ordering over the integers provided by absolute value

The typical ordering has no least or greatest element; for every integer, you can find one that is higher in the ordering, and one that is lower. But in the absolute value order, there is a specific least element; 0. (And, of course, no greatest element.)

You can get pretty crazy with this. Here are some more examples.

[TODO just definitely pictures for all of these]

The natural numbers with the typical ordering, except you declare that 0 is the greatest element.
The integers ordered by 1x
Collate the natural numbers by their lowest divisor. E.g. everything divisible by 2 is lower than everything divisible by 3 (but not 2) and 2 is still lower than 4.

This is a way in which orders can have fundamentally different structures; two orders with the same "shape" can have a bijection (i.e. a relabeling of states) between them such that the ordering is preserved, but orders with different shapes do not have such a bijection.

Minimizers vs maximizers

An ordering either has a least element, or it does not have a leas...
AI Safety via Luck
by Jozdien — 2023-04-01T20:13:55.346Z

Epistemic Status: I feel confident and tentatively optimistic about the claims made in this post, but am slightly more uncertain about how it generalizes. Additionally, I am concerned about the extent to which this is dual-use for capabilities and

exfohazardous

and spent a few months thinking about whether it was worth it to release this post regardless. I haven’t come to an answer yet, so I’m publishing this to let other people see it and know what they think I should do.

TL;DR: I propose a research direction to solve alignment that potentially doesn’t require solutions to

ontology identification

, learning how to

code

, or becoming

literate

.

Introduction

Until a few hours ago, I was spending my time primarily working on

high-level interpretability

and

cyborgism

. While I was writing a draft for something I was working on, an activity that usually yields me a lot of free time by way of procrastination, I stumbled across the central idea behind many of the ideas in this post. It seemed so immediately compelling that I dropped working on everything else to start working on it, culminating after much deliberation in the post you see before you.

My intention with this post is to provide a definitive reference for what it would take to safely use AGI to steer our world toward much better states in the absence of a solution to any or all of several existing problems, such as

Eliciting Latent Knowledge

,

conditioning simulator models

,

Natural Abstractions

,

mechanistic interpretability

, and the like.

In a world with prospects such as those, I propose that we radically rethink our approach to AGI safety. Instead of dedicating enormous effort to engineering nigh-impossible safety measures, we should consider thus-far neglected avenues of research, especially ones that have memetic reasons to be unfairly disprivileged so far and which immunizes them against capabilities misuse. To avert the impending AI apocalypse, we need to focus on high-variance, low-probability-high-yield ideas: lightning strikes that, should they occur, effectively solve astoundingly complex problems in a single fell swoop. A notable example of this, which I claim we should be investing all of our efforts into, is luck. Yes, luck!

Luck As A Viable Strategy

I suggest that we should pay greater attention to luck as a powerful factor enhancing other endeavors a...
Quaker Practice for the Aspiring Rationalist
by maia — 2023-04-01T19:32:03.051Z

I wrote this up for a meetup I ran recently, where we held a Quaker-style Meeting for Worship session. I recommend doing this in a group setting, but you can also do it by yourself if the ideas appeal to you.

Why Quaker meeting

I’ve had depressive episodes throughout my life and one of the few things that seems to consistently help is going to Quaker Meeting.

Some of this may be the community (religious practice of any kind is supposed to be helpful). But I’ve found it useful even when I participated very little in the community around it, even just doing Quaker practice by myself.

I don’t think this is true of everyone who does it. But it works for me, and I have some idea why. For this meetup, I want to try to convey some of that to you, and teach you how to sit in silence in a way that makes your life better.

I have an ulterior motive here, which is that while I do care about and enjoy the Quaker community, there are certain deep divides between me and it that sometimes make it hard to feel comfortable. I think I’m more in sync with most rationalists, and I imagine there are parts of Quaker practice that would be a lot easier to do with a community I feel more generally at home in. So I’m hoping I can get some of you interested in doing this more.

What we’ll be doing:

Explanation of what I do during Meeting
Questions, discussion, etc.
Queries
Silent Practice
Debrief & food

The usual Quaker advice

Disclaimer: There are lots of different explanations or suggestions for what to do during Meeting for Worship. Quaker practice has a long history and I’m frankly not an expert in it. But I went to Quaker schools growing up and developed my own approach, based on the stuff my teachers and other adults told me. I’m going to share that approach here.

Second disclaimer: This is NOT meditation. There are definitely some similarities, especially in the “settling in” part that I’m going to describe, but the important parts are pretty different, from what I have read. (However: Quaker meeting is ultimately a practice of “sit in silence” and I can’t police what’s going on in your head! And plenty of Quakers historically and currently have done stuff more akin to meditation. But it’s not what I’m going to tell you about.)

General Quaker advice is that there is “the light of God in everyone”, and that during Meeting for Worship, your goal is to listen to that “inner light.” Or to list...
The Plan: Put ChatGPT in Charge
by Sven Nilsen — 2023-04-01T17:23:07.980Z

Problem:
1. AI is becoming too smart to control
2. Sooner or later, we will have an AI takeover
Solution:
Put an AI that is too stupid to take over the world by itself, but trained on ethical data, in charge to control the AI takeover safely.
If you ask ChatGPT ethical questions to be answered "Yes", "No" or "STOP" if the question is too complex, then you will find that it is more ethical than many world leaders today.
The obvious solution is to put world leaders below ChatGPT in political power. Develop a safe protocol for asking ChatGPT ethical questions and make it possible to verify the answer.
Discussion?
AI infosec: first strikes, zero-day markets, hardware supply chains, adoption barriers
by Allison Duettmann — 2023-04-01T16:44:01.592Z

[This part 1 of a 5 part sequence on security and cryptography areas relevant for AI safety, published and linked here a few days apart]

AI safety in practice

AI safety in practice relies on the AI system not only being aligned, but also not being able to discover internal computer security vulnerabilities itself or be stealable by attackers. The latter problem may be harder than one would assume at first sight. Here are a few reasons why that is, discussed at length by Mark S. Miller, Christine Peterson, and I in the

Defend Against Cyber Threats

chapter in Gaming the Future:

First Strike Instabilities

In

Information security considerations for AI and the long term future

,

Jeffrey Ladish and Lennart Hein anticipate intense competition surrounding the development of AGI leading to considerable interest from state actors. They assign a high likelihood to advanced threat actors targeting organizations involved in AGI development, supply of critical resources to AGI companies, or possession of strategically important information, in order to gain an advantage in AGI development.

In

Defend Against Cyber Threats

in Gaming the Future, we suggest that this threat may be worse than often acknowledged, due to ‘first strike instabilities’: if the possibility of an AGI takeover merely becomes credible and believed to be imminent, this poses a significant risk in itself. If one nation-state actor becomes aware that another actor is about to develop an AGI capable of taking over the world, they may choose to preemptively destroy it. Critically, this risk exists even if creating an AGI is impossible, but simply believed to be possible.

It’s possible the attacker merely tries to steal the AGI capability or destroy the AGI-creating actor rather than launch a nation-wide attack. But it’s likely that the nation hosting the AGI-creating actor would retaliate, and the attacking nation knows this. Given how high the stakes may be believed to be at that point, the attacker may not want to risk this. We live in a world where multiple militaries have nuclear weapon delivery capabilities, so public AGI arms races may preemptively re-create game theoretic dynamics that resemble the Cold War nuclear game theory.

If a cyber-attack is chosen by the attacker, it would introduce several relatively new characteristics compared to a kinetic attack:

severe cyber attacks may already

...
Introducing AlignmentSearch: An AI Alignment-Informed Conversional Agent
by BionicD0LPH1N — 2023-04-01T16:39:09.643Z

Authors: Henri Lemoine, Thomas Lemoine, and Fraser Lee

We are excited to introduce

AlignmentSearch

, an attempt to create a conversational agent that can answer questions about AI alignment. We built this site in response to ArthurB’s

$5k bounty for a LessWrong conversational agent

calling for the creation of a chatbot capable of discussing bad AI Alignment takes, and guiding people new to the field through common misunderstandings and early points of confusion.

Tl;dr

AlignmentSearch uses a dataset about AI alignment to construct a prompt for ChatGPT to answer AI alignment-related questions, while citing established sources. We qualitatively observe a massive boost in the quality of answers over ChatGPT without any specialized prompt, with results either on par or better than those given by much stronger LLMs (GPT-4 and Bing Search).

Two answers backed by ChatGPT

Overview

AlignmentSearch indexes the

Alignment Research Dataset

, generating vector embeddings to enable nearest-semantic-neighbor search. We take a user query to find the top-k most semantically close “paragraphs” (text blocks of around 220 tokens, plus some padding). The size of the dataset means we generally have access to a few paragraphs that are very semantically similar to the user’s question, and are likely to contain information that is useful in producing an answer. We form a prompt that gives these paragraphs to ChatGPT along with the user’s question and instructions on citation, and retrieve an answer somewhere between summary and synthesis, all with accurate inline citations linking to the source material. We’ve created a website that lets the user interact easily with this process: ask questions, dive into the sources used in an answer, and ask for further clarifications. We also have an alternative mode that exposes the raw results of the semantic search.

Alignment Research Dataset

The alignment research dataset was announced

on LessWrong

in June 2022, with a related paper accessible

here

. It contains posts and papers from a wide range of websites and books about alignment, including LessWrong, stampy.ai, arXiv, the alignment newsletter, etc. As it was created in June ‘22, recent posts are absent. We are in the process of updating the dataset to contain recent AI alignment content.

Variable Source Quality

Not all sources are made equal. Posts on LessW...