trentmkelly/LessWrong-43k huggingface dataset:
Common misconceptions about OpenAI
by Jacob_Hilton — 2022-08-25T14:02:26.257Z

I have recently encountered a number of people with misconceptions about OpenAI. Some common impressions are accurate, and others are not. This post is intended to provide clarification on some of these points, to help people know what to expect from the organization and to figure out how to engage with it. It is not intended as a full explanation or evaluation of OpenAI's strategy.

The post has three sections:

Common accurate impressions
Common misconceptions
Personal opinions

The bolded claims in the first two sections are intended to be uncontroversial, i.e., most informed people would agree with how they are labeled (correct versus incorrect). I am less sure about how commonly believed they are. The bolded claims in the last section I think are probably true, but they are more open to interpretation and I expect others to disagree with them.

Note: I am an employee of OpenAI. Sam Altman (CEO of OpenAI) and Mira Murati (CTO of OpenAI) reviewed a draft of this post, and I am also grateful to Steven Adler, Steve Dowling, Benjamin Hilton, Shantanu Jain, Daniel Kokotajlo, Jan Leike, Ryan Lowe, Holly Mandel and Cullen O'Keefe for feedback. I chose to write this post and the views expressed in it are my own.

Common accurate impressions

Correct: OpenAI is trying to directly build safe AGI.

OpenAI's

Charter

states: "We will attempt to directly build safe and beneficial AGI, but will also consider our mission fulfilled if our work aids others to achieve this outcome." OpenAI leadership describes trying to directly build safe AGI as the best way to currently pursue OpenAI's mission, and have expressed concern about scenarios in which a bad actor is first to build AGI, and chooses to misuse it.

Correct: the majority of researchers at OpenAI are working on capabilities.

Researchers on different teams often work together, but it is still reasonable to loosely categorize OpenAI's researchers (around half the organization) at the time of writing as approximately:

Capabilities research: 100
Alignment research: 30
Policy research: 15

Correct: the majority of OpenAI employees did not join with the primary motivation of reducing existential risk from AI specifically.

My strong impressions, which are not based on survey data, are as follows. Across the company as a whole, a minority of employees would cite reducing existential risk from AI as their top reason for joining. A ...
Your posts should be on arXiv
by JanB — 2022-08-25T10:35:12.087Z

TL;DR: There are many posts on the Alignment Forum/LessWrong that could easily be on arXiv. Putting them on arXiv has several large benefits and (sometimes) very low costs.

Benefits of having posts on arXiv

There are several large benefits of putting posts on arXiv:

1. Much better searchability, shows up in google scholar searches.
2. Additional reads (arXiv sanity, arXiv newsletters, and so on).
3. The article can accumulate citations, which are shown in google/google scholar search results.

1) - 3) lead to more people reading your research, which hopefully leads to more people building on it and maybe useful feedback from outside of the established alignment community. In particular, if people see that the paper already has citations, this will lead to more people reading it, which will lead to more citations, and so on.

You'd gain even more of 2) and 3) from publishing it at a conference, but, unlike arXiv, that's significant additional work (often still worth it).

There are also some smaller benefits from publishing on arXiv:

4. firmly establishes this as your contribution (not sure, but I think if you only have an alignment forum post, someone could build a bit on it and then claim the whole thing as their contribution because alignment forum posts don't count?).
5. better citability (e.g. if somebody writes an ML paper to be published in ML venues, it gives more credibility to cite arXiv papers than Alignment Forum/LessWrong posts. The same goes for people writing e.g. Wikipedia articles about alignment.)

How much work is it to submit to arXiv?

Citing DavidHolmes from the comments: "There is a certain amount of moderation on arXiv. This is a little opaque [...]. In writing this I don't want to give the impression that posting things to arXiv is hard; I have currently 28 papers there, have never had a single problem or delay with moderation, and the submission process generally takes me <15 mins these days."

Sometimes, I think getting your forum post ready for submission can be as easy as creating a pdf of your post (although if your post was written in LaTeX, they'll want the tex file). If everything goes well, the submission takes less than an hour.

However, if your post doesn't look like a research article, you might have to format it more like one (and even then it's not guaranteed to get in, see this comment thread).

If you are submitting to ...
The Solomonoff prior is malign. It's not a big deal.
by Charlie Steiner — 2022-08-25T08:25:56.205Z

Epistemic status: Endorsed at ~85% probability. In particular, there might be clever but hard-to-think-of encodings of observer-centered laws of physics that tilt the balance in favor of physics. Also, this isn't that different from Mark Xu's post.

Previously, previously, previously

I started writing this post with the intuition that the Solomonoff prior isn't particularly malign, because of a sort of pigeon hole problem - for any choice of universal Turing machine there are too many complicated worlds to manipulate, and too few simple ones to do the manipulating.

Other people have different intuitions.

So there was only one thing to do.

Math.

All we have to do is compare [wild estimates of] the complexities of two different sorts of Turing machines: those that reproduce our observations by reasoning straightforwardly about the physical world, and those that reproduce our observations by simulating a totally different physical world that's full of consequentialists who want to manipulate us.

Long story short, I was surprised. The Solomonoff prior is malign. But it's not a big deal.

Team Physics:

If you live for 80 years and get 10^7 bits/s of sensory signals, you accumulate about 10^16 bits of memory to explain via Solomonoff induction.

In comparison, there are about 10^51 electrons on Earth - just writing their state into a simulation is going to take somewhere in the neighborhood of 10^51 bits[1]. So the Earth, or any physical system within 35 orders of magnitude of complexity of the Earth, can't be a Team Physics hypothesis for compressing your observations.

What's simpler than the Earth? Turns out, simulating the whole universe. The universe can be mathematically elegant and highly symmetrical in ways that Earth isn't.

For simplicity, let's suppose that "I" am a computer with a simple architecture, plus some complicated memories. The trick that allows compression is you don't need to specify the memories - you just need to give enough bits to pick out "me" among all computers with the same architecture in the universe[2]. This depends only on how many similar computers there are in Hilbert space with different memories[3], not directly on the complexity of my memories themselves.

So the complexity of a simple exemplar of Team Physics more or less looks like:

Complexity of our universe

+ Complexity of my architecture, and rules for reading observations out ...
[Review] The Problem of Political Authority by Michael Huemer
by Arjun Panickssery — 2022-08-25T05:18:51.883Z
Link: https://arjunpanickssery.substack.com/p/review-the-problem-of-political-authority

The state is often ascribed a special sort of authority, one that obliges citizens to obey its commands and entitles the state to enforce those commands through threats of violence. This book argues that this notion is a moral illusion: no one has ever possessed that sort of authority.
...

Anarchists face a catch-22: most people will not give anarchism a serious hearing because they are convinced that the position is crazy; they are convinced that the position is crazy because they do not understand it; and they do not understand it because they will not give it a serious hearing.
— Michael Huemer

Bryan Caplan, who went to Berkeley with Huemer, writes:

I’ve read almost every major work of libertarian political philosophy ever published. In my view, Michael Huemer’s new The Problem of Political Authority: An Examination of the Right to Coerce and the Duty to Obey is the best book in the genre.

Michael Huemer’s genius is that he so often presents the kind of arguments that aren’t on your radar at all but are obvious as soon as they’re presented to you. The Problem of Political Authority is like that. Unlike Nozick or other deontological libertarians, he doesn’t begin with any strong premises about property rights. He just begins with common-sense beliefs:

A presumption against coercion—People shouldn’t attack, kill, kidnap, steal from, or defraud other people outside of a few special circumstances (like with consent, or to prevent those other people from harming third parties, or to prevent much worse consequences).
A recognition that the government is coercive—While direct physical violence isn’t a routine punishment, the threat of violence underpins the incremental series of penalties that governments do routinely impose. For example, drivers who run red lights on empty streets might be fined $200, but if they repeatedly fail to pay, their driver’s licenses may be revoked, and if they continue to drive anyway, they may be ultimately imprisoned (kidnapped?).
The imposition of the burden of proof on the government—If the state claims to have an authority to do what ordinary individuals can’t, then this claim has to be justified by some special property of the state per se.

The first two are clear, and the second is introduced with a story (Nozick tells a similar story in Anarchy, State, and Utopia called “The Tale of the Slave”) about a random vigilante who resolves to ...
The Shard Theory Alignment Scheme
by David Udell — 2022-08-25T04:52:50.206Z

Generated as part of SERI MATS, Team Shard, under John Wentworth.

Thanks to Logan Smith, Garrett Baker, Michael Einhorn, Quintin Pope, and Thomas Kwa for chatting about related topics. All mistakes are my own.

The Alignment Scheme

The shard theory alignment scheme is:

Master the theory of value formation in trained intelligences, and develop powerful chain-of-thought interpretability tools to examine those trained values in action. Use that understanding and those interpretability tools to instill a target value (e.g., corrigibility, niceness, or libertarianism) in a powerful language model.

Then, punt the remainder of the alignment problem to researchers equipped with that aligned powerful language model.

…easier said than done, of course! In particular, the "master the theory of value formation in trained intelligences" and "develop powerful chain-of-thought interpretability tools" steps together contain almost all of the original alignment problem! So, I'll briefly elaborate on Team Shard's approach to both below.

Understand the Phenomenon of Value Formation inside Trained Intelligences

What this premise in the scheme demands is a completed mechanistic theory of algorithm formation in trained intelligences, conditional on various training variables being set in different ways. This … is a lot to ask of any plucky team of alignment researchers, and is the sort of demand that many an alignment researcher would reflexively glance off of. It's one of the unavoidable core difficulties of aligning ML systems, though -- it's something we'll have to have in all the worlds where ML alignment succeeds.

We conjecture that reinforcement strengthens the behavior-steering computations that guide a system into reinforcement events, and that those behavior-steering computations will only form around abstractions already represented inside of a system at the time of reinforcement. We bet that there are a bunch of quantitative relationships here just waiting to be discovered -- that there's a lot of systematic structure in what learned values form given which training variables. To ever get to these quantitative relationships, we'll need to muck around with language model fine-tuning under different conditions a lot.

So, mucking around with running pilot experiments on large language models in controlled environments (RL text adventures!) is what we're doing now! In particular, we'r...
Preparing for the apocalypse might help prevent it
by Ocracoke — 2022-08-25T00:18:58.832Z

When I tell people that I think there is a decent chance that an unaligned AGI will bring about the apocalypse within the next 20 years or so, they tend to not take it too seriously. Often that's because they think I would act differently if I really assigned a high probability to this.

When I say apocalypse, I personally consider it more likely that the development of AGI will lead to major disruptions like war, civil unrest or breakdown of supply chains or basic infrastructure. I can also imagine that an AGI actively tries to kill all humans, but I consider that less likely. Even if it actively tried to disempower humans, it wouldn't need to kill most of us, it would just have to destroy or control our institutions or infrastructure.

Because of that, I think it would be prudent to learn how to survive without supply chains, electricity, and so on. Basically, become a prepper. Not only would that increase the chances of my own survival in a range of different scenarios, it would also be an honest signal to others that I'm serious about the potential threat. I would be putting my money where my mouth is.

I'm surprised that despite the high probability that many people here assign to AGI takeover scenarios, there is little talk about how we might improve our survival chances if we can't avoid bad outcomes. I suspect that this is partially because being a prepper has certain connotations like being a conspiracy theorist and having wonky ideas. AGI takeover is also a wonky idea, but a different kind of wonky, and we don't want to be grouped together with people we think shouldn't be taken seriously.
So maybe the problem with this approach is that AGI worries would be taken less seriously by others.

Or maybe the reason why there isn't much talk about preparing for bad AGI outcomes is that there is a sense that this can wait until the threat is more imminent?
What Makes A Good Measurement Device?
by johnswentworth — 2022-08-24T22:45:06.421Z

In some sense, any new measurement device automates a part of research. A thermometer automates the task of sticking one's finger in something to check how hot it is, a scale automates the task of holding something to check how heavy it is, etc. The automated version is not only more convenient, but more precise and reproducible (as is usually the case when automating things). For alignment, one analogue might be interpretability tools which automate the work done when some human looks at a part of a neural net and sees what it's doing.

Let's take that last example and dig into it a bit more: interpretability tools which automate the work done when some human looks at a part of a net and sees what it's doing. We want to leverage the analogy to other measurement tools, like a thermometer or a scale, to better understand automation of interpretability.

Here's one type of proposal I hear a lot: to automate interpretability, have some human researchers look at parts of a net, poke at them, and write up an explanation of how they're interpreting it. Collect data from many such instances, and train a neural net to take net-parts and produce explanations.

We want to leverage the analogy to thermometers or scales, so what would be the analogous strategy for making a thermometer or scale? Well, have a bunch of humans stick their fingers in a bunch of stuff and report how hot the stuff is, then train a neural net to replicate the humans' hotness-reports. Or, have a bunch of humans hold things and report how heavy they are, then train a net to replicate the humans' heaviness-reports.

Hopefully it is obvious that the "train a net to replicate human reports" results would not be nearly as useful, for purposes of scientific progress, as actual thermometers or scales. But what's missing? And how can we carry that insight back to the interpretability problem?

The thermometer has two great powers: a simple legible data type, and reproducilbility. First, simple legible data type: the thermometer's output is a single number (the temperature), and we can compare that number with other thermometer-outputs. That's a kind-of-thing for which we have very precise mathematical understanding: we know exactly what kinds-of-things we can do with these numbers, we have a nice general representation, we're confident that different people mean the same thing by numbers, etc. This is in co...
Consciousness Actually Explained: EC Theory
by Casey_ — 2022-08-24T22:18:05.022Z
Link: https://proteanbazaar.substack.com/p/consciousness-actually-explained

Greetings!

This has taken too long to make, and I'd like to polish it more, but it's time to release and move on to other things. Apologies for not posting the text here, I was having formatting trouble.

Yes, I attempt to solve/explain consciousness. The main document is around 13,000 words long, which takes an hour to skim-read (not recommended) but probably 2+ hours to read correctly.

Let me know what you think!

Edit 9/9/23: Of the criticisms I've seen of EC (both here and elsewhere), none have struck me as having passed my ITT, and/or there is frequently some selective amnesia on display (or more probably, people skim-read the document) regarding points I explicitly take great effort to address. One might wonder why I largely haven't engaged with EC's critics, since consciousness can correctly be considered the most important feature of the universe; important to get right. But in light of the solution to C, comparatively little actually hinges/depends on C, or on what people think of it. When combined with the subtle conceptual nature of the problem itself (lossless communication is difficult and idiosyncratic), these factors have lead me to largely not deem it worth my energy to engage. If someone had an actually novel or ITT-passing criticism, I would be in a better position to engage. Lacking that, the most compact and well formulated memetic package on offer is still the document itself. If I was invested in any given critic "getting it", then my engagement would largely look like me insisting they re-read the document with greater care, and/or I would go through it line by line with them, to combat the amnesia problem and actually find where they go off the rails. That kind of effort doesn't scale.
Google AI integrates PaLM with robotics: SayCan update [Linkpost]
by Evan R. Murphy — 2022-08-24T20:54:34.438Z
Link: https://sites.research.google/palm-saycan

Back in April, Google AI announced SayCan, a project which integrated a language model (FLAN) with robotics in order to produce a robot which could follow instructions. For example, cleaning up a mess in a kitchen. (LessWrong post from April which links to a tweet about SayCan.)

This week Google AI has released some new updates, dubbed PaLM-SayCan. This involved upgrading the integrated language model to Google AI's top performing large language model (LLM) of 540-billion parameters, PaLM.

Website for PaLM-SayCan

Blog post announcing PaLM-SayCan

The following updates from this week are quoted from the original SayCan project website (linked in first sentence of this post, bold mine):

[8/16/2022] We integrated SayCan with Pathways Language Model (PaLM), and updated the results. We also added new capabilities including drawer manipulation, chain of thought prompting and multilingual instructions. You can see all the new results in the updated paper.
[8/16/2022] Our updated results show that SayCan combined with the improved language model (PaLM), which we refer to as PaLM-SayCan, improves the robotics performance of the entire system compared to a previous LLM (FLAN). PaLM-SayCan chooses the correct sequence of skills 84% of the time and executes them successfully 74% of the time, reducing errors by a half compared to FLAN. This is particularly exciting because it represents the first time we can see how an improvement in language models translates to a similar improvement in robotics.
[8/16/2022] We open-sourced a version of SayCan on a simulated tabletop environment.

Thanks to Jon Menaster for making me aware of this update from Google AI.
History of Zero-based Months?
by jefftk — 2022-08-24T20:30:01.865Z

JavaScript is a language of many silly things, and one of them is:

> new Date()
Wed Aug 24 2022 ...
> new Date().getFullYear()
2022
> new Date().getMonth()
7
> new Date().getDate()
24

It represents 2022-08-24 as (2022, 7, 24). One-based indexing for the
year and day, but zero-based indexing for the month.

In this case, however, the problem was copied from
Java:

getMonth:
The value returned is between 0 and 11, with the value 0 representing
January.

getDate:
The value returned is between 1 and 31 representing the day of the
month.

I'd love to blame Java, but they seem to have copied the problem from C:

localtime(3):

tm_mday: The day of the month, in the range 1 to 31.
tm_mon: The number of months since January, in the range 0 to 11.

Looking at the Unix History
repo, the first
mention of "month (0-11)" is in 1973's Research Unix
V4:

The value is a pointer
to an array whose components are
.s3
.lp +5 5
0 seconds
.lp +5 5
1 minutes
.lp +5 5
2 hours
.lp +5 5
3 day of the month (1-31)
.lp +5 5
4 month (0-11)
.lp +5 5
5 year \*- 1900
.lp +5 5
6 day of the week (Sunday = 0)
.lp +5 5

While this may have been an original decision by Dennis
Ritchie, it's also possible it was copied from an even earlier
system. Does anyone know?

Comment via: facebook