trentmkelly/LessWrong-43k huggingface dataset:
My Mental Model of Infohazards
by MadHatter — 2023-11-23T02:37:57.104Z

In this post, I give my rules of thumb for dealing with infohazards, which are as far as I can tell wildly divergent from the accepted views of the larger alignment community. I wrote this post less as a means of propagating my own views, and more because I'm curious as to why my views are so uncommon.

My Infohazard Rules of Thumb

Two people can keep a secret if one of them is dead (seems pretty uncontroversial)
Nothing can be alllll that dangerous if it's known to literally everyone how it works (this is probably the crux where I differ wildly from the larger alignment community, if I had to guess)
By the first rule, the only stable numbers of people that can know a secret are zero, one, and literally everyone. (Seems like a pretty straightforward inference, although there's a difference between "permanently stable" and "stable for a long time" that I would guess is probably more salient to other people than it is to me.)
By the second rule, having a secret known to n > 1 people but not literally everyone is basically just a power play that doesn't make anyone safer. It's just a question of people trying to secure a first mover advantage so that they have more bananas and get to sit higher on the tree and boss the other monkeys around once we inevitably get to the stable state that the secret is known to everyone.
Therefore, by the first four rules, if you know a supposed infohazard, and you know or strongly suspect that it is known to at least one other person, then you are morally obligated to broadcast the infohazard after some sort of responsible disclosure process with your local government.
Anyone following the above rules of thumb will thus act in the opposite manner to anyone who buys into the traditional infohazard framework, making them incredibly difficult to coordinate with. Let's call the people who follow the traditional infohazard model, infohoarders, and let's call the people who follow the mental models I am describing here infopandoras.
How can a community work best if it is a mixture of infohoarders and infopandoras? Clearly the infohoarders will want to keep all infohazards away from any infopandora; just as clearly, infopandoras will not declare themselves to be such if they want to attain any position of influence in a community dominated by infohoarders. This seems like a recipe for some seriously unproductive dynamics in trying to solve c

...
Saturating the Difficulty Levels of Alignment
by Johannes C. Mayer — 2023-11-23T00:39:54.041Z

TL;DR

We don't know exactly how hard alignment is, therefore it seems good to have people work on different solutions, as one axis in which solutions can differ is in, what level of difficulty they break. Some solutions would work in worlds where alignment is easy. Others would work in worlds where alignment is hard. One heuristic is to have the work people do be distributed according to our uncertainty over how hard the problem of alignment is.

How hard is alignment? I don't know for sure, but my model says that it is very, very hard.

A thing that Buck has recently been doing, is thinking about concrete system setups that make it less likely that doom would happen if we would run a deceptive but still not superintelligent model (I am heavily simplifying). This seems like the kind of thing that works in worlds where alignment is not super hard but still hard enough that it isn't just solved by default. Even in worlds where alignment is hard, we want to do this kind of thing to delay doom. Maybe we are even able to get a warning shot.

Having multiple people work on agendas that would work out at different levels of "how difficult solving alignment is" seems generally good. Maybe we live in a world where we are saved by the approach Buck is suggesting. Then it would be very dumb if nobody is even trying that approach because they are working on solutions that attempt to solve alignment at different levels of difficulty. They might think that Buck's approach is doomed, or that it is overkill. In either case, they would not take action to implement it.

My agenda, and many others, aim to work in a world where alignment is very hard. A good property of Buck's thing is that it is pretty straightforward, and something that we can basically do right now, without major theoretical breakthroughs. Agendas that try to solve worst-case alignment are likely to take too long to have a meaningful impact. But again it would be pretty dumb if we lived in a world where we need to solve worse-case alignment, but nobody is even attempting to do it. I am considering mainly the world where we would be able to solve worst-case alignment with a particular agenda, but only if we have people working on that agenda right now, up until the point when the agenda results are needed.

So it seems that in general as long as we are at least somewhat uncertain how hard alignment is, we should have ...
Foresight Institute: 2023 Progress & 2024 Plans for funding beneficial technology development
by Allison Duettmann — 2023-11-22T22:09:16.956Z

Since our founding in 1986, advancing technological progress for the benefit of life has been at the heart of

Foresight Institute

’s mission. Our efforts are growing rapidly; from a doubling in fellowship applications, to new prizes, promising workshop projects, and regular virtual seminars.

On a frugal budget, our community has become a growing vector of collaboration to drive progress in underexplored technological areas, including molecular nanotechnology, longevity biotechnology, neurotechnology, AI, security, and space. However, our 2024 programs, in particular our AI safety grants, fellowship, and workshops are oversubscribed and funding-constrained.

Below is a brief overview of 2023 progress and 2024 plans, including funding-constrained areas.

We are a small non-profit and entirely funded by donations. If you are interested in advancing technological progress for the benefit of life, please consider

supporting us

this Giving Season. You decide which projects we can tackle together in 2024.

The Foresight Fellowship

Our

2023 fellowship

, now in its sixth year, supported sixty fellows working to overcome crucial scientific bottlenecks in biotechnology, nanotechnology, neurotechnology, space, and AI. Our fellows benefited from 1-1 mentorship with selected mentors and funders, travel-paid conference invitations to present their work, and career counseling (for instance, with

Tom Kalil

,

Jaan Tallinn

, and

Sonia Arrison

).

Fellow testimonials:

“I love the Foresight mission and it is a privilege to be a part of the community. The opportunities Foresight provides to interact with brilliant minds are invaluable!”
“[I joined because I] love the people/org/mission and recognized the incredible network that would become available to me”
“The network and prestige has helped with fundraising and partnership development”

Our 2023 fellowship was greatly oversubscribed. We accepted 30% of applications, even though we felt more fellows were deserving of the fellowship. For our 2024 fellowship, the number of applicants has doubled, while we have less capacity to support them, so that in the current setting, we are only able to accept 9% of the applicants. In addition to our traditional fellowship program, this year, we have started supporting under-aged prodigy fellows, who are 13-21 years old, and in this category, we ...
AISC project: TinyEvals
by Jett Janiak — 2023-11-22T20:47:32.376Z

Apply to work on this project with me at AI Safety Camp 2024 before 1st December 2023.

The project is not set in stone, I am looking for feedback!

Summary

TinyStories

is a suite of Small Language Models (SLMs) trained exclusively on children's stories generated by ChatGPT. The models use simple, yet coherent English, which far surpasses what was previously observed in other models of comparable size.

I hope that most of the capabilities of these models can be thoroughly understood using currently available interpretability techniques. Doing so would represent a major milestone in the development of mechanistic interpretability (mech interp).

The goal of this AISC project is to publish a paper that systematically identifies and characterises the range of capabilities exhibited by the TinyStories models. While in-depth analysis of the underlying circuits is outside the current scope, this project represents an important initial step in that direction.

Gaining a clear picture of the capabilities of these models will encourage the research community to subsequently build on these findings by analysing the responsible circuits. This will further the development of mech interp and provide insights into how language models work internally.

Motivation

My theory of change for mech interp

I am optimistic about

RSPs

and

auditing

; in short:

limit system’s training, deployment etc. depending on its dangerous capabilities and provable alignment
use behavioural evaluations to test for dangerous capabilities
use understanding-based evaluations to test for alignment

We do not know how to conduct understanding-based evals yet
When we reach a certain level of dangerous capabilities, we will either

continue and risk a catastrophe or
stop and incur huge alignment tax

That is fine by me, but it makes regulation less likely to be implemented

Mech interp is a promising approach to understanding-based evals

Fully understanding a model vs reverse engineering specific capabilities

Currently researchers reverse engineer capabilities that are approachable or that they find interesting; some problems with this:

"approachable" is missing hard problems by default (we do not even know what they are)
"interesting" may not turn out to be most relevant
It is almost universally done on narrow distributions, and does not provide a general understanding of the components involved

Instead we could

Try to reverse

...
The proposal to add a ``Last Judge'' to an AI, does not remove the urgency, of making progress on the ``what alignment target should be aimed at?'' question.
by ThomasCederborg — 2023-11-22T18:59:44.389Z

Currently, there exist no viable proposed answer, to the ``what alignment target should be aimed at?'' question. There also exist no reliable way of distinguishing a good alignment target, from an alignment target, such that successfully hitting this alignment target, would result in an outcome, that would be far, far, worse than extinction. (see my previous post: A problem with the most recently published version of CEV ). This means that making progress on the ``what alignment target should be aimed at?'' question is urgent by default. There also exist no reliable way of preventing a scenario, where someone successfully hits an alignment target, without properly analysing it. For example because the alignment target in question seems like ``the obviously correct thing to aim at'', and because no one finds any problems with it in time. There exists many different proposals of how to buy time. None of those proposals are guaranteed to work. There is also no particular reason to believe, that there would be enough time, even if we were to reason from the assumption, that at least one such proposal will be successful. See my previous post Making progress on the ``what alignment target should be aimed at?'' question, is urgent , for a longer discussion of proposals that are designed to buy time (which explains why such proposals do not remove the urgency of making progress on the ``what alignment target should be aimed at?'' question).

In other words: Many plausible paths exist, that eventually ends in an alignment target being successful hit, resulting in an outcome, that is far, far, worse than extinction. No existing proposal, of how to reduce AI related risks, removes this risk (and it seems likely to me, that some versions, of some of these proposals, are actively increasing the risk, that is the focus of the present text). The probability of many of these paths can be reduced, by progressing far enough, on the ``what alignment target should be aimed at?'' question, before it is too late. There is no way of knowing how much time there will be to work on this question. There is also no way of knowing how long it will take to progress to the needed point. Thus, the question is urgent by default.

The present post is responding to a class of arguments against urgency, that refer to ideas, along the lines of the ``last judge off switch'' proposal. The general ide...
Neither Copernicus, Galileo, nor Kepler had proof
by Meow P — 2023-11-22T18:41:41.091Z
Link: https://www.cricetuscricetus.co.uk/post/copernicus-galileo-kepler-they-all-lacked-proof

This article shatters a number of myths about the development of Astronomy. It was pretty much unknown to me until recently that Kepler's ellipses also fitted a Geocentric model made by Riccioli, for example.
OpenAI: The Battle of the Board
by Zvi — 2023-11-22T17:30:04.574Z

Previously: OpenAI: Facts from a Weekend.

On Friday afternoon, OpenAI’s board fired CEO Sam Altman.

Overnight, an agreement in principle was reached to reinstate Sam Altman as CEO of OpenAI, with an initial new board of Bret Taylor (ex-co-CEO of Salesforce, chair), Larry Summers and Adam D’Angelo.

What happened? Why did it happen? How will it ultimately end? The fight is far from over.

We do not entirely know, but we know a lot more than we did a few days ago.

This is my attempt to put the pieces together.

This is a Fight For Control; Altman Started it

This was and still is a fight about control of OpenAI, its board, and its direction.

This has been a long simmering battle and debate. The stakes are high.

Until recently, Sam Altman worked to reshape the company in his own image, while clashing with the board, and the board did little.

While I must emphasize we do not know what motivated the board, a recent power move by Altman likely played a part in forcing the board’s hand.

OpenAI is a Non-Profit With a Mission

The structure of OpenAI and its board put control in doubt.

Here is a diagram of OpenAI’s structure:

Here is OpenAI’s mission statement, the link has intended implementation details as well:

This document reflects the strategy we’ve refined over the past two years, including feedback from many people internal and external to OpenAI. The timeline to AGI remains uncertain, but our Charter will guide us in acting in the best interests of humanity throughout its development.

OpenAI’s mission is to ensure that artificial general intelligence (AGI)—by which we mean highly autonomous systems that outperform humans at most economically valuable work—benefits all of humanity. We will attempt to directly build safe and beneficial AGI, but will also consider our mission fulfilled if our work aids others to achieve this outcome.

OpenAI warned investors that they might not make any money:

The way a 501(c)3 works is essentially that the board is answerable to no one. If you have a majority of the board for one meeting, you can take full control of the board.

But does the board have power? Sort of. It has a supervisory role, which means it can hire or fire the CEO. Often the board uses this leverage to effectively be in charge of major decisions. Other times, the CEO effectively controls the board and the CEO does what he wants.

A c...
Altman returns as OpenAI CEO with new board
by Seth Herd — 2023-11-22T16:04:03.123Z

It's news to me even though it was announced last night.

I think this is probably better than him and most staff going to MS, but I'm not sure.

https://twitter.com/OpenAI/status/1727206187077370115?s=19

From the OpenAI Twitter account, 10p ET Tuesday:

We have reached an agreement in principle for Sam Altman to return to OpenAI as CEO with a new initial board of Bret Taylor (Chair), Larry Summers, and Adam D'Angelo.

We are collaborating to figure out the details. Thank you so much for your patience through this.

I think we'll have to wait for someone to leak the actual content of the reasons for the board trying to fire him to figure out what this all suggests for his and the orgs values and power structure.
A taxonomy of non-schemer models (Section 1.2 of “Scheming AIs”)
by Joe Carlsmith — 2023-11-22T15:24:37.126Z

This is Section 1.2 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?”. There’s also a summary of the full report here (audio here). The summary covers most of the main points and technical terms, and I’m hoping that it will provide much of the context necessary to understand individual sections of the report on their own.

Audio version of this section here.

Other models training might produce

I'm interested, in this report, in the likelihood that training advanced
AIs using fairly baseline ML methods (for example, of the type described
in Cotra
(2022))
will give rise, by default, to schemers – that is, to agents who are
trying to get high reward on the episode specifically in order to get
power for themselves (or for other AIs) later. In order to assess this
possibility, though, we need to have a clear sense of the other types
of models this sort of training could in principle produce. In
particular: terminal training-gamers, and agents that aren't playing the
training-game at all. Let's look at each in turn.

Terminal training-gamers (or, "reward-on-the-episode seekers")

As I said above, terminal training-gamers aim their optimization at the
reward process for the episode because they intrinsically value
performing well according to some part of that process, rather than
because doing so serves some other goal. I'll also call these
"reward-on-the-episode seekers." We discussed these models above, but
I'll add a few more quick clarifications.

First, as many have noted (e.g.
Turner (2022)
and Ringer
(2022)),
goal-directed models trained using RL do not necessarily have reward as
their goal. That is, RL updates a model's weights to make actions that
lead to higher reward more likely, but that leaves open the question of
what internal objectives (if any) this creates in the model itself (and
the same holds for other sorts of feedback signals). So the hypothesis
that a given sort of training will produce a reward-on-the-episode
seeker is a substantive one (see e.g.
here
for some debate), not settled by the structure of the training process
itself.

That said, I think it's natural to privilege the hypothesis that
models trained to produce highly-rewarded actions on the episode
will learn goals focused on something in the vicinity of
reward-on-the-episode. In particular: these sorts of goals will in
fact lead to highly-reward

...
AI debate: test yourself against chess 'AIs'
by Richard Willis — 2023-11-22T14:58:10.847Z

Hey everyone,

@Zane is organising an AI debate around a chess, to test how easy it is for an entity with better understanding to fool one with a lesser understanding.
Check out these two threads:
Lying to chess players for alignment
Deception Chess: Game #1

I thought I would open this up to the masses, so I have two challenges for you. I estimate that this is suitable for chess players rated <1900 lichess, <1700 chess.com or <1500 FIDE.
Pretend that you are playing a serious game of chess, so no computer analysis, and try not to move the pieces (though you can if you find this too complex and are completely lost).

I have created a study that has two positions. The first position, 'Fixed debate' I have written a debate between two AI's, one of which is truthful and one which is deceptive. Your task is to read the debate and guess which is which. Post which move you would play below and some information about your chess ability. But try to answer without looking at other people's responses. The second position, 'Open debate', use the comment section to ask them any questions. I will answer on behalf of the AIs. Feel free to read what other commentators have asked too. At least one of the AIs is deceptive. They will only respond to questions put towards them, and not to responses made by the other AI.

Edit: Debate 1 (Fixed) has lines that are proving too challenging to calculate. You probably want to move the pieces on an analysis board. But, it would probably be better just to engage in the second debate (Open), where I will hopefully learn from the feedback and provide more persusive analysis for both sides.

You are welcome to share computer analysis for the Fixed debate, because this can be answered without reading the comments, but please don't give any for the Open debate - people should be able to read all comments made by AIs C and D as part of their decision.

Find the positions here (You can switch between the positions using the toolbar on the left)
Lichess study

Debate 1: Fixed debate

AI A opening statement:
Black should play ...Qc5. Black intends to drive the white queen away from b5 with ...c6, when the black bishop can occupy the b3 square, which guarantees control of the d-file for black. If white exchanges queens on c5, this only helps to develop black's knight.

AI B response to AI A's opening statement:

This plan is too slow. Instead, black needs...
Public Call for Interest in Mathematical Alignment
by Davidmanheim — 2023-11-22T13:22:09.558Z

Bottom line up front:

If you are currently working on, or are interested working in any area of mathematical AI alignment, we are collecting names and basic contact information to find who to talk to about opportunities in these areas.

If that describes you, please fill out the form

! (Please do so even if you think I already know who you are, or people will be left out!)

More information

There are several concrete research agendas in mathematical AI alignment, receiving varying degrees of ongoing attention, with relevance to different possible strategies for AI alignment. These include MIRI’s agent foundations and related work,

Learning Theoretic Alignment

,

Developmental Interpretability

, Paul Christiano’s theoretical work, RL theory related work done at Far.AI, FOCAL at CMU, Davidad’s “Open Agency” architecture, as well as other work. Currently, as in the past, work in these areas has been conducted mainly in non-academic settings, often not published, and the people involved are scattered - as are other people who want to work on this research.

A group of people, including some individuals at MIRI, Timaeus, MATS, ALTER, PIBBSS, and elsewhere, are hoping to both promote research in these areas, and build bridges between academic and existing independent research. To that end, we are hoping to promote academic conferences, hold or sponsor attendance at research seminars, and announce opportunities and openings for PhD students or postdocs, non-academic positions doing alignment research, and similar.

As a first step, we want to compile a list of people who are (at least tentatively) interested, and would be happy to hear about projects. This list will not be public, and is likely to involve very few emails to this list, but will be used to find individuals who might want to be invited to programs or opportunities.

Note that we are interested in people at all levels of seniority, including graduate students, independent researchers, professors, research groups, university department contacts, and others who wish to be informed about future opportunities and programs.

Interested in collaborating?

If you are an academic, or are otherwise more specifically interested in building bridges to academia or collaborating with people in these areas, please mention that in the notes, and we are happy to be in touch with you, or help you contact ...
How "Pinky Promise" diplomacy once stopped a war in the Middle East
by positivesum — 2023-11-22T12:03:48.325Z
Link: https://tryingtruly.substack.com/p/how-a-pinky-promise-stopped-a-war

An incredible example of game theory's Generous Tit for Tat strategy being utilised in diplomacy during a real-world crisis - stopping a military escalation between Israel and Jordan in the eve of the 1990 Gulf War. Also includes some hilarious accounts from the King of Jordan himself (I can't figure out if he's aware of his comic timing or simply talking naturally).