trentmkelly/LessWrong-43k huggingface dataset:
Sequence Reread: Fake Beliefs [plus sequence spotlight meta]
by Raemon — 2022-11-02T00:09:11.755Z

A couple months ago the LessWrong team started putting Sequence Spotlights at the top of the home page, to give people reasonably bite-sized chunks of canonical LessWrong content to read (or reread). i.e, this thing here:

I thought I'd experiment with making a post where people could discuss the posts, whether you're reading them for the first time or coming back to them.

The Fake Beliefs sequence introduces a bunch of ways that "things we call beliefs" can, well, not actually be beliefs. Noticing you don't actually believe a thing is a foundational rationalist skill.

Object level discussion prompts:

What does it feel like to have a "fake belief"?
What notable fake beliefs have you found since getting into "sequences-style-rationality?". Were there any subtler fake beliefs that took longer to notice?
If you first read this awhile ago, has your perspective on fake beliefs changed?
If you think someone's belief is fake, and they don't share a bunch of rationality culture, what's the best way to engage them?

Meta level discussion prompts:

How are you generally feeling about Sequence Spotlights?

What is your experience of them like? Have you clicked on anything?
Do you get value from seeing art or canonical reading on the top of the page, independent from reading it?

I've considered giving Sequence Spotlights a "Create Discussion Meetup" button, such that Sequence Spotlights more easily serve as default reading material for local communities. Would anyone be interested in that?
Real-Time Research Recording: Can a Transformer Re-Derive Positional Info?
by Neel Nanda — 2022-11-01T23:56:06.215Z
Link: https://youtu.be/yo4QvDn-vsU

New experiment: Recording myself real-time as I do mechanistic interpretability research! I try to answer the question of what happens if you train a toy transformer without positional embeddings on the task of "predict the previous token" - turns out that a two layer model can rederive them! You can watch me do it here, and you can follow along with my code here. This uses a transformer mechanistic interpretability library I'm writing called EasyTransformer, and this was a good excuse to test it out and create a demo!

This is an experiment in recording and publishing myself doing "warts and all" research - figuring out how to train the model and operationalising an experiment (including 15 mins debugging loss spikes...), real-time coding and tensor fuckery, and using my go-to toolkit. My hope is to give a flavour of what actual research can look like - how long do things actually take, how often do things go wrong, what is my thought process and what am I keeping in my head as I go, what being confused looks like, and how I try to make progress. I'd love to hear whether you found this useful, and whether I should bother making a second half!

Though I don't want to overstate this - this was still a small, self-contained toy question that I chose for being a good example task to record (and I wouldn't have published it if it was TOO much of a mess).
All AGI Safety questions welcome (especially basic ones) [~monthly thread]
by Robert Miles — 2022-11-01T23:23:04.146Z

tl;dr: Ask questions about AGI Safety as comments on this post, including ones you might otherwise worry seem dumb!

Asking beginner-level questions can be intimidating, but everyone starts out not knowing anything. If we want more people in the world who understand AGI safety, we need a place where it's accepted and encouraged to ask about the basics.

We'll be putting up monthly FAQ posts as a safe space for people to ask all the possibly-dumb questions that may have been bothering them about the whole AGI Safety discussion, but which until now they didn't feel able to ask.

It's okay to ask uninformed questions, and not worry about having done a careful search before asking.

Stampy's Interactive AGI Safety FAQ

Additionally, this will serve as a way to spread the project Rob Miles' volunteer team[1] has been working on: Stampy - which will be (once we've got considerably more content) a single point of access into AGI Safety, in the form of a comprehensive interactive FAQ with lots of links to the ecosystem. We'll be using questions and answers from this thread for Stampy (under these copyright rules), so please only post if you're okay with that! You can help by adding other people's questions and answers to Stampy or getting involved in other ways!

We're not at the "send this to all your friends" stage yet, we're just ready to onboard a bunch of editors who will help us get to that stage :)

Stampy - Here to help everyone learn about stamp maximization AGI Safety!

We welcome feedback[2] and questions on the UI/UX, policies, etc. around Stampy, as well as pull requests to his codebase. You are encouraged to add other people's answers from this thread to Stampy if you think they're good, and collaboratively improve the content that's already on our wiki.

We've got a lot more to write before he's ready for prime time, but we think Stampy can become an excellent resource for everyone from skeptical newcomers, through people who want to learn more, right up to people who are convinced and want to know how they can best help with their skillsets.

PS: Based on feedback that Stampy will be not serious enough for serious people we built an alternate skin for the frontend which is more professional: Alignment.Wiki. We're likely to move one more time to aisafety.info, feedback welcome.

Guidelines for Questioners:

No previous knowledge of AGI safety is required. If y

...
Which Issues in Conceptual Alignment have been Formalised or Observed (or not)?
by ojorgensen — 2022-11-01T22:32:25.243Z

One aspect of alignment research I find interesting is improving how we understand potential threat / failure modes by making them more concrete or rigorous in some way. When I think of making these concrete, I’m thinking of theoretical results in ML (think Turner’s power-seeking work) or toy examples of the behaviour arising (that funny boat doing reward hacking).

To my knowledge, some conceptual failure modes which are central to the field have not been observed yet (deceptive alignment and gradient hacking come to mind). This isn't to suggest that they won't be observed in more powerful systems, but it seems important to note nonetheless.

I think it would be useful to have a spreasheet of major conceptual failure modes, alongside whether they have been observed in the wild / formalised in ML, and any mathematical formalisations / results surrounding them.

With a view to potentially making this table, what conceptual failure modes do you know of that have yet to be “formalised or observed”? I'd be particularly interested to hear about any absence of evidence which has surprised you in some way.

Conversely, what are some formalisations / theoretical results you are aware of for some conceptual failure mode?
AI as a Civilizational Risk Part 4/6: Bioweapons and Philosophy of Modification
by PashaKamyshev — 2022-11-01T20:50:54.078Z

Bio-Chemical Weapons made by AI

One of the core risks involved with both narrow AIs and general AIs is the creation of biological and chemical weapons. While the probability of creating stronger and stronger behavior modification in AIs is close to 100%, the probability of creating biological weapons using AI is smaller. However, the potential downsides are more significant. A larger probability is that AIs create many chemical quasi-weapons compounds that can end up in manufacturing due to a misunderstood safety profile.

Analogizing signal pollution to manufacturing pollution is worth considering, especially because we have yet to solve manufacturing pollution. There are many credible hypotheses

that many chemicals have polluted the land of industrial civilizations.

Some chemicals may cause

hormonal disruptions

in people and animals. One of the challenges is that

chemicals are usually considered innocent until proven guilty

. A key civilizational challenge is to cost-effectively ascertain the guilt or innocence of chemicals or their minimum dangerous levels. Creating cost-effective chemical profiles is an old challenge. However, it is suitable as a metaphor for determining the extent of "signal pollution" or minimum safe levels of disconnect between a meme and its meaning.

However, this is not just important to think about as a metaphor. As AI chemical and biological manufacturing ramps up, it will create new compounds or increase some of the concentrations of existing ones. Without a proper pipeline for deciding how to deal with novel chemicals, this presents a challenge. Having chemicals considered innocent until proven guilty is wrong. Most biological modifications to people are likely harmful.

Overuse of chemicals is already causing problems.

However, having each chemical go through 10 years of safety checks might also need to be corrected if information can be obtained faster about its relative safety. For example, if someone finds less toxic plastic, putting it into production faster to replace other plastic is better, even if it is not 100% safe. Humanity needs to solve this challenge should be solved at the meta-level with AIs working on new chemical compounds having proper controls down to the utility function level.

However, the situation worsens when we have committed actors using AI...
Open & Welcome Thread - November 2022
by MondSemmel — 2022-11-01T18:47:40.682Z

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the Highlights from the Sequences, a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the Concepts section.

The Open Thread tag is here. The Open Thread sequence is here.
Mildly Against Donor Lotteries
by jefftk — 2022-11-01T18:10:06.458Z

Let's say you're going to donate some money, and plan to put some time
into figuring out the best place to donate. The more time you put
into your decision the better it's likely to be, but at some point you
need to stop looking and actually make the donation. The more you're
donating the longer it makes sense to go, since it's more valuable to
better direct a larger amount of money. Can you get the benefits of the
deeper research without increasing time spent on research? Weirdly,
you can! About five years ago, Carl Schulman

proposed
(and

ran the first) "donor lottery".

Say instead of donating $1k, you get together with 100
other people and you each put in $1k. You select one of the people at
random (1:100) to choose where the pool ($100k) goes. This turns your
100% chance of directing $1k into a 1% chance of directing $100k. The
goal is to make research more efficient:

If you win, you're working with enough money that it's worth it
for you to put serious time into figuring out your best donation option.

If you lose, you don't need to put any time into determining where
your money should go.

This isn't that different from giving your money to GiveWell or similar to decide how
to distribute, in that it's delegating the decision to someone who can put
in more research. Except that it doesn't require identifying someone
better at allocating funds than you are, it just requires that you
would:

Make better decisions in allocating $100k than $1k

Prefer a 1% chance of a well-allocated $100k to a 100% chance of a
somewhat less well-allocated $1k.

If you come at this from a theoretical perspective this can seem
really neat: better considered donations, more efficient use of
research time, strictly better, literally no downsides, why would you
not do this?

Despite basically agreeing with all of the above, however, I've not
participated in a donor lottery, and I think they're likely slightly
negative on balance. There actually are downsides, just not
in areas that the logic above considers.

The biggest downside is that it makes your donation decisions less
legible: it's harder for people to understand what you're doing and
why. Lets say you're talking to a friend:

Friend: You're really into charity stuff, right? Where did you decide to
donate this year?

You: I put my money into a donor lottery. I trade

...
Progress links and tweets, 2022-11-01
by jasoncrawford — 2022-11-01T17:48:45.562Z
Link: https://rootsofprogress.org/links-and-tweets-2022-11-01

Announcements

The Reef Starter Innovation Challenge, to identify startups that could benefit from operations in Low Earth Orbit
AI startup Generally Intelligent has launched and open-sourced their research environment (via @kanjun)
Deep Science Ventures is launching a doctorate to put venture creation at the heart of the PhD (via @isabel_thomp)
Interintellect launches a fellowship/microgrant program (via @TheAnnaGat)
At Story Summit, founders will come together to communicate the futures they see in three immersive days of writing (via @AlexandriaLabs)
Ben Reinhardt seeking help to build a website for his private ARPA project (@Ben_Reinhardt)
The Foresight Institute is hiring an executive assistant (via @allisondman)
Prediction market Metaculus launches a “Forecasting Our World In Data” tournament (via @metaculus)

Links

Progress is driven by impatient optimists by Hannah Ritchie (via @BillGates!)
How to escape scientific stagnation in The Economist, Matt Clancy is quoted (h/t @heidilwilliams_)
Works in Progress special issue on abundance and the technological frontier
A proposed agenda for the “abundance movement” (by @s_r_constantin). See also her follow up list of policy organizations
Why wasn’t the steam engine invented earlier? (by @antonhowes)
Improving the culture and social processes of science (by @michael_nielsen and @kanjun)
For the first time ever, humans changed the motion of a celestial object (via @NASA)
The Long Now republishes my essay “Can Growth Continue?”, based on my 5-minute Ignite talk (via @longnow)
The breast cancer treatment that got funding from the billionaire owner of Revlon (Virginia Postrel)
A NEPA explainer from @elidourado. Related, 3 years and $1.7M to build a single public toilet in SF
Exposure to markets makes people behave more cooperatively and in less self-interested ways (Alex Tabarrok)
22nd cohort Emergent Ventures includes Jackson Oswalt, who achieved nuclear fusion at age 12 (yes, this is a thing)
Vox announced the Future Perfect 50 list of people building a better future. I’m on it (via @voxdotcom and @bryanrwalsh)

Queries

How should creatives respond to the advance of AI?
Best thinkers/writing on how we can evolve the NRC to help us build more nuclear? (@juliadewahl)
Good blog or Twitter feed of Notion power user tips?
Books that examine the drivers and limits of technological innovation? (@brunoswerneck)
Good histories of how governments fig

...
On the correspondence between AI-misalignment and cognitive dissonance using a behavioral economics model
by Stijn Bruers — 2022-11-01T17:39:10.433Z

This essay is one of the awarded essays for the Econ Theory AI Alignment Prize from Superlinear. The essay uses the tools of theoretical economics to contribute to the AI alignment problem.

Psychologists are familiar with the notion of cognitive dissonance, the discomfort when a person’s behavior conflicts with that person’s beliefs and preferences. Cognitive dissonance is accompanied with seemingly irrational behavior such as motivated reasoning and self-deception. It is basically an example of misalignment: decisions are not aligned with values. In that sense, perhaps cognitive dissonance can shed some light on the AI-alignment problem (Bostrom, 2014): how to create safe artificial intelligence that does things that we (creators, humans, sentient beings) really want? How can the decisions of AI-machines become aligned with our values and preferences? Even a small misalignment between our values and the goals of superintelligent machines (that are more intelligent and hence powerful than us) could cause very serious problems. And we will never be intelligent enough to solve those problems once the misaligned machines are more intelligent than us.

Computer scientists who study the AI-alignment problem speak a different language than psychologists who study cognitive dissonance. But new studies in behavioral economics, especially some new theoretical models about cognitive dissonance (Hestermann, Le Yaouanq & Treich, 2020), can perhaps bridge this gap between computer scientists and psychologists. The language of economists, when they speak of e.g. utility functions, optimization, expected value maximization, game-theoretic strategic behavior and Bayesian updating, is closer to the language of computer scientists who develop AI.

The cognitive dissonance models in behavioral economics are well illustrated by the meat paradox (Hestermann, Le Yaouanq & Treich, 2020): many people are animal-loving meat eaters, and their high levels of meat consumption are not aligned with their concern for animal welfare. They do not want to cause unnecessary animal suffering, but they know that meat consumption involves unnecessary suffering when there are alternatives (e.g. plant-based meat) that cause less or no suffering. When those people do not switch to a meat-free diet, they start rationalizing their meat consumption, denying a mind to farm animals (Basti...
Threat Model Literature Review
by zac_kenton — 2022-11-01T11:03:22.610Z

TL;DR: This post provides a literature review of some threat models of how misaligned AI can lead to existential catastrophe. See our accompanying post for high-level discussion, a categorization and our consensus threat model.

Where available we cribbed from the summary in the Alignment Newsletter.

For other people's overviews of some threat models, see here and here.

[ETA: When considering strengths and weaknesses of each threat model, this was done with respect to our goal of better generation/prioritization among alignment research projects. They shouldn't necessarily be read as an all-things-considered review of that work.]

Categorization

See our accompanying post for an explanation of this categorization

Threat Models

Goal misgeneralization + Specification Gaming ➜ MAPS

Is Power-Seeking AI an Existential Risk?

(Carlsmith)

Summary

This report investigates the classic AI risk argument in detail, and decomposes it into a set of conjunctive claims. Here’s the quick version of the argument: We will likely build highly capable and agentic AI systems that are aware of their place in the world, and which will be pursuing problematic objectives. Thus, they will take actions that increase their power, which will eventually disempower humans, leading to an existential catastrophe. We will try and avert this, but will probably fail to do so since it is technically challenging and we are not capable of the necessary coordination.

There’s a lot of vague words in the argument above, so let’s introduce some terminology to make it clearer:

- Advanced capabilities: We say that a system has advanced capabilities if it outperforms the best humans on some set of important tasks (such as scientific research, business/military/political strategy, engineering, and persuasion/manipulation).

- Agentic planning: We say that a system engages in agentic planning if it (a) makes and executes plans, (b) in pursuit of objectives, (c) on the basis of models of the world. This is a very broad definition and doesn’t have many of the connotations you might be used to for an agent. It does not need to be a literal planning algorithm -- for example, human cognition would count, despite (probably) not being just a planning algorithm.

- Strategically aware: We say that a system is strategically aware if it models the effects of gaining and maintaining power over humans and the real-world environment...
Clarifying AI X-risk
by zac_kenton — 2022-11-01T11:03:01.144Z

TL;DR: We give a threat model literature review, propose a categorization and describe a consensus threat model from some of DeepMind's AGI safety team. See our post for the detailed literature review.

The DeepMind AGI Safety team has been working to understand the space of threat models for existential risk (X-risk) from misaligned AI. This post summarizes our findings. Our aim was to clarify the case for X-risk to enable better research project generation and prioritization.

First, we conducted a literature review of existing threat models, discussed their strengths/weaknesses and then formed a categorization based on the technical cause of X-risk and the path that leads to X-risk. Next we tried to find consensus within our group on a threat model that we all find plausible.

Our overall take is that there may be more agreement between alignment researchers than their

disagreements

might suggest, with many of the threat models, including our own consensus one, making similar arguments for the source of risk. Disagreements remain over the difficulty of the alignment problem, and what counts as a solution.

Categorization

Here we present our categorization of threat models from our literature review, based on the technical cause and the path leading to X-risk. It is summarized in the diagram below.

In green on the left we have the technical cause of the risk, either specification gaming (SG) or goal misgeneralization (GMG). In red on the right we have the path that leads to X-risk, either through the interaction of multiple systems, or through a misaligned power-seeking (MAPS) system. The threat models appear as arrows from technical cause towards path to X-risk.

The technical causes (SG and GMG) are not mutually exclusive, both can occur within the same threat model. The distinction between them is motivated by the common distinction in machine learning between failures on the training distribution, and when out of distribution.

To classify as specification gaming, there needs to be bad feedback provided on the actual training data. There are many ways to operationalize good/bad feedback. The choice we make here is that the training data feedback is good if it rewards exactly those outputs that would be chosen by a competent, well-motivated AI[1]. We note that the main downside to this operationalisation is that even if just one out o...