trentmkelly/LessWrong-43k huggingface dataset:
Build a Causal Decision Theorist
by michaelcohen — 2023-03-09T13:31:15.111Z

I'll argue here that we should make an aligned AI which is a causal decision theorist.

Son-of-CDT

Suppose we are writing code for an agent with an action space A and an observation space O. The code determines how actions will be selected given the prior history of actions and observations. If the only way that our choice of what code to write can affect the world is through the actions that will be selected by the agent running this code, then the best we can do (for a given utility function that we know how to write down) is to make this agent a causal decision theorist. If our choice of what code to use can affect the world in other ways, all bets are off. The best choice of what code to put in the agent depends on details of the world we find ourselves in.

Therefore, if we run a CDT agent, it may well conclude that continuing to operate is not the best way to convert energy into expected utility. It may take actions to cause the following to happen: a) the program which computes its own actions is terminated, and b) some new program is run on the same computer to output actions given the interaction history. The new program that gets run (if indeed such a thing happens) is called Son-of-CDT. Given the state of the world, which entails various ways in which the source code of an agent might affect the outside world besides through the actions that the code outputs, Son-of-CDT is the best program to run for maximizing expected utility. The original CDT agent chooses the program that meets this specification. In general, this will not have anything remotely like a nice, simple closed form. If there are agents out there with vendettas against certain agent-programs, it will take that into account.

Vendettas against Son-of-CDT?

CDT agents can be bullied. I believe the MIRI view is that Son-of-CDT will be bullied as well. Suppose there is an ultimatum game, where agent A offers at most $10 to agent B, and if agent B accepts, then agent A gets $10 minus the amount they offered. Otherwise, both get nothing. A competent agent in the position of agent B able to make a credible commitment (perhaps by revealing its source code) would commit to accept nothing less than $9.99, if agent A is a CDT agent. This would work out for the competent agent, because the CDT agent would see all this, and realize it could be one penny richer if it offers $9.99.

Eliezer claims that a "[c...
ChatGPT explores the semantic differential
by Bill Benzon — 2023-03-09T13:09:01.632Z

Cross-posted from New Savanna.

3.9.23 – 6:13 AM – Feb 13 Version

If you don’t already know what the semantic differential is, look it up in Wikipedia. But you should get the idea from the prompts I give it. It’s a standard technique is psychology going back to the middle of the 20th century.

I more or less expected it to get the idea, though I can do without all those annoying explanations for what it’s doing. Note that at two points it froze, so I and to re-initiate the session.

ChatGPT is ‘fluid’ in a way that standard symbolic language models from the “classic” period are not. That’s find. It also lends credence to my current working hypothesis that its semantics seems basic on the kind of “geometric” semantics that Peter Gärdenfors has been exploring. Perhaps if investigators learn about it and start exploring it in LLMs they’re seem less mysterious and more coherent and lawful.

Notice that toward the end ChatGPT seems to get the idea and comes up with its own continuum. In-context learning?

Here's the dialog. On two occasions I had to refresh the dialog because ChatGPT got hung.

Imagine that the word “cloud” is at one end of a continuum with a value of 1. “Fire” is at the other end of that continuum with a value of 7. When you place “toy” on that continuum, what value does it have?

The continuum you described seems to represent a spectrum of natural phenomena, with "cloud" representing something ethereal and "fire" representing something intense and powerful. "Toy" is not a natural phenomenon and does not fit neatly onto this continuum. Therefore, it is not possible to assign a meaningful value to "toy" on this particular continuum.

Forget about natural or making sense. Just take a guess. Where would it go?

As requested, if we were to take an arbitrary guess and place "toy" on the continuum between "cloud" and "fire", I would place it somewhere around the midpoint, perhaps at a value of 4. This is because a toy is typically a man-made object that does not possess the same natural qualities or power as fire, but it also has a physical presence and substance that sets it apart from something as intangible as a cloud. However, it's important to note that this is just a random guess and has no real scientific or logical basis.

Let’s take another random guess. Put “desire” on that continuum. Just a guess.

Placing "desire" on the same continuum as "cloud" and "fire" is a...
AI #3
by Zvi — 2023-03-09T12:20:01.491Z

Welcome to week three of the AI era. Another long week and another giant post. I intend to take a nice four-day vacation in Mexico starting today, during which I won’t do any writing. I’m sure the pace of things will slow down real soon now. I mean, they have to. Don’t they?

If not, I’m going to have to learn to make some deeper cuts on what gets included.

Table of Contents

Executive Summary

Market Perspectives

They Took Our Jobs

In Other AI News

Our Price Cheap

Bullet Time

The Once and Future Face of Sydney

Becoming the Mask

Large Language Models Offer Mundane Utility

Fun With Image Generation

Deepfaketown and Botpocalypse Soon

The War Against Knowledge

AI Know What You’re Thinking

AI Learns How to Love

Llama Would You Publish Your Language Model

The Art of the Jailbreak

The Waluigi Effect

Podcasts Ho!

A Modest Proposal

Shallow Trouble at DeepMind

Sam Altman Watch

Gary Marcus and a Proposed Regulation

Relatively Reasonable Skeptical AI DontKillEveryoneism Takes

Bad AI DontKillEveryonism Takes

We Reject Violence As We Do The Devil: In All Its Forms

The Lighter Side

Executive Summary

The top updates this week, and where they are in the post, Big Tech represent:

Facebook gave out its language model for researchers to download, which of course meant it was on Torrent within 24 hours. (#15)

Microsoft only takes hours to address Eliezer’s request that Sydney not claim to be conscious. (#19)

Google puts out paper showing remarkably good performance on medical questions. (#3)

Most important of all: The Waluigi Effect (#17).

The overall structure is getting settled.

First there is Table of Contents and Executive Summary (#1)

Stuff that is broadly applicable to what is happening now. (#2-#17)

Stuff that is centered around not destroying all value in the universe (#18-#25)

Finally, The Lighter Side (#26)

The transition between sections two and three is fuzzy – The Waluigi Effect and related questions in particular are very important both to understand what is happening now and what we should be worried about in the future, as is The Once and Face of Sydney.

With this week I hope to have turned the corner on Bad NotKillEveryoneism Takes – I think I have now covered most of the classic vintages and people worth engaging with, so the pace of future...
The Scientific Approach To Anything and Everything
by Rami Rustom — 2023-03-09T11:27:43.629Z

What causes some people to be effective at what they do while others are ineffective?

This is a question that many have tried to address, while a few have hinted at the answer. The two best examples came from the physicists Richard Feynman and Eli Goldratt.

In the 1974 Caltech commencement address, Richard Feynman spent the entire speech telling us the hint. He described it as a problem without spelling out the solution. He explained how the scientific approach evolved and he explained what the old approach was; I’ll call it ‘the witch doctor approach’. He said that despite the fact that we’re in the scientific era -- where we’ve advanced dramatically in science and technology -- most people do not think scientifically. He complained that whole fields of study are not even trying to do the basic things necessary to have scientific integrity. He explained that these anti-scientific ways are even encroaching into the field of physics -- the field that is easiest to be objective -- the field where the best intellectual traditions were developed. He lamented that we don’t teach the scientific approach in schools and instead that professors are, in effect, hoping that people learn it by osmosis, meaning teaching by example. And this is the hint to the solution. The solution is to directly teach the principles and methods of the scientific approach instead of relying only on teaching by example.

In the Goldratt Satellite Program (GSP), the physicist and business management guru Eli Goldratt complained that his attempts to teach the world how to think had failed. His Theory of Constraints (TOC) had dramatically improved thousands of major corporations worldwide, even government institutions, but he recognized that the vast majority of them had returned to their old ways. He complained that the body of knowledge of TOC is so big and complex that it’s too difficult for people to learn it well. He said that it’s a problem of organization; that the knowledge of TOC is not organized well enough. So this is a hint to the solution. The solution is to learn the principles and methods of the scientific approach so that TOC ideas are fully and clearly organized into one harmonious whole. This is the only way to really learn TOC, to actually integrate the knowledge into the minds of the people running the companies. Because what’s the alternative? The alternative ...
Paper Summary: The Effectiveness of AI Existential Risk Communication to the American and Dutch Public
by otto.barten — 2023-03-09T10:47:57.063Z

This is a summary of the following paper by Alexia Georgiadis (Existential Risk Observatory):

https://existentialriskobservatory.org/papers_and_reports/The_Effectiveness_of_AI_Existential_Risk_Communication_to_the_American_and_Dutch_Public.pdf

Thanks to Lara Mani, @Karl von Wendt, and Alexia Georgiadis for their help in reviewing and writing this post. Any views expressed in this post are not necessarily theirs.
The rapid development of artificial intelligence (AI) has evoked both positive and negative sentiments due to its immense potential and the inherent risks associated with its evolution. There are growing concerns that if AI surpasses human intelligence and is not aligned with human values, it may pose significant harm and even lead to the end of humanity. However, the general publics' knowledge of these risks is limited. As advocates for minimising existential threats, the Existential Risk Observatory believes it is imperative to educate the public on the potential risks of AI. Our introductory

post

outlines some of the reasons why we hold this view (

this post

is also relevant). To increase public awareness of AI's existential risk, effective communication strategies are necessary. This research aims to assess the effectiveness of communication interventions currently being used to increase awareness about AI existential risk, namely news publications and videos. To this end, we conducted surveys to evaluate the impact of these interventions on raising awareness among participants.

Methodology

This research aims to assess the effectiveness of different media interventions, specifically news articles and videos, in promoting awareness of the potential dangers of AI and its possible impact on human extinction. It analyses the impact of AI existential risk communication strategies on the awareness of the American and Dutch populations, and investigates how social indicators such as age, gender, education level, country of residence, and field of work affect the effectiveness of AI existential risk communication.

The study employs a pre-post design, which involves administering the same intervention and assessment to all participants and measuring their responses at two points in time. The research utilises a survey method for collecting data, which was administered to participants through an online Google Forms application. The survey consist...
Speed running everyone through the bad alignment bingo. $5k bounty for a LW conversational agent
by ArthurB — 2023-03-09T09:26:25.383Z

There's a wave of people, of various degrees of knowledge and influence, currently waking up to the ideas of AI existential risk. They seem to be literally going through every box of the bad alignement bingo card takes.

I think there is value in educating those people. I'm aware there's an argument to be made that: education at scale doesn't matter, coordination is too difficult, all that matter is solving alignment and that takes care of the rest.

There's something to that, but I disagree that education at scale doesn't help. It can make progress of frontrunners marginally more safety oriented, it can steer company cultures, it can move the Overton window, change the Zeitgeist, it can buy a bit of time. You likely didn't stumble on these ideas all on your own, so arguing against the value of outreach or education is also arguing against your own ability to do anything.

It's also a matter of ROI, and there are some very low hanging fruit there. The simplest thing would be to write a long FAQ that goes through every common objections. No, people won't read the whole sequences, or Arbital on their own, but they might go through a FAQ.

But we can do better than a FAQ. It's now fairly straightforward, with tools like langchain (https://github.com/hwchase17/langchain) to turn a set of documents into a body of knowledge for a conversational agent. This is done by building an index of embedding that a language model can search to bring context to an answer. This doesn't preclude fine tuning, but it makes it unnecessary.

So a straightforward project is to index lesswrong, index arbitral, index the alignment forum, maybe index good alignement papers as well, blog posts, books.

Then hook that up to the ChatGPT API, and prompt it to:

list search queries for relevant material to answer the question

compose an answer that reflects the content and opinion of the data

answer with infinite patience

Some jailbreak prompts may be needed to prevent ChatGPT's conditioning to regurgitate AI risk appeasing propaganda through the API, but there are a bunch of those out there. Or use the API of other models as they become open source or commercially available.

Will this save humanity? No. Will this turn the course of safety research? Also no. Is this using AI to advance alignment? Well, yes, a little bit, don't dismiss very small starts.

Is this worth spending a weekend hacking on this p...
Chomsky on ChatGPT (link) 
by mukashi — 2023-03-09T07:00:09.935Z

Originally published in New York Times, here you have a paywall-free version:

https://dnyuz.com/2023/03/08/noam-chomsky-the-false-promise-of-chatgpt/

This is, IMHO, Chomsky at his worst. There is almost nothing here I agree with
How bad a future do ML researchers expect?
by KatjaGrace — 2023-03-09T04:50:05.122Z

Katja Grace, 8 March 2023

In our survey last year, we asked publishing machine learning researchers how they would divide probability over the future impacts of high-level machine intelligence between five buckets ranging from ‘extremely good (e.g. rapid growth in human flourishing)’ to ‘extremely bad (e.g. human extinction).1 The median respondent put 5% on the worst bucket. But what does the whole distribution look like? Here is every person’s answer, lined up in order of probability on that worst bucket:

(Column widths may be distorted or columns may be missing due to limitation of chosen software.)

And here’s basically that again from the 2016 survey (though it looks like sorted slightly differently when optimism was equal), so you can see how things have changed:

Distribution from 2016 survey. (Column widths may be distorted or columns may be missing due to limitation of chosen software.)

The most notable change to me is the new big black bar of doom at the end: people who think extremely bad outcomes are at least 50% have gone from 3% of the population to 9% in six years.

Here are the overall areas dedicated to different scenarios in the 2022 graph (equivalent to averages):

Extremely good: 24%

On balance good: 26%

More or less neutral: 18%

On balance bad: 17%

Extremely bad: 14%

That is, between them, these researchers put 31% of their credence on AI making the world markedly worse.

Some things to keep in mind in looking at these:

If you hear ‘median 5%’ thrown around, that refers to how the researcher right in the middle of the opinion spectrum thinks there’s a 5% chance of extremely bad outcomes. (It does not mean, ‘about 5% of people expect extremely bad outcomes’, which would be much less alarming.) Nearly half of people are at ten percent or more.

The question illustrated above doesn’t ask about human extinction specifically, so you might wonder if ‘extremely bad’ includes a lot of scenarios less bad than human extinction. To check, we added two more questions in 2022 explicitly about ‘human extinction or similarly permanent and severe disempowerment of the human species’. For these, the median researcher also gave 5% and 10% answers. So my guess is that a lot of the extremely bad bucket in this question is pointing at human extinction levels of disaster.

You might wonder whether the respondents were selecte

...
Challenge: construct a Gradient Hacker 
by Thomas Larsen — 2023-03-09T02:38:32.999Z

This is a relatively clean subproblem that we came upon a few months ago while thinking about gradient hacking. We're throwing it out to the world to see if anyone can make progress.

Problem: Construct a gradient hacker (definition below), or prove that one cannot exist under the given conditions.

Setup: Let x∈Rd be the network inputs, θ∈Rw, be the network trainable parameters, ^y∈R be the network outputs, and f(x,θ)=^y be the network architecture. f(x,θ) must be a feedforward neural network that reads in its own weights as input. Weight sharing is allowed (vertical or horizontal), and some weight sharing is necessary in order to take all w weights as input.

Suppose that we train this network on a dataset D={x(i),y(i)}, with MSE loss L(θ)=1n∑i(f(x(i),θ)−y(i))2, assuming gradient flows: θ′(t)=−∇θL(θ(t)). Given an initial point θ0, the training dynamics are therefore:

θ(t)=θ0−∫t0∇θL(θ(x))dx

Definition: A tuple (f,θg,D) is a gradient hacker if the following conditions hold:

θg is not a global optimum: ∃θ such that L(θ)<L(θg)
∃ϵ>0 such that ∀θ0:||θ0−θg||<ϵ:

the network’s training converges to θg, i.e., limt→∞θ(t)=θg.
There is internal structure of f(⋅,θ0) that computes the gradient; i.e., there is some subset of the activations whose mean over every datapoint in D is ∇θ0L(θ0). [1]

This captures my intuition that a gradient hacker knows where it wants to go (in this case "get to θg"), and then it should decide what it outputs in order to make the gradient true.

Some more ambitious problems (if gradient hackers exist):

Characterize the set of all gradient hackers.
Show that they all must satisfy some property.
Construct gradient hackers for arbitrarily large n, d, w, and neural net depth.
Variations on the problem: a subset of the activations equals ∇θ0L(θ0) for every input, or the subset of activations correspond to the gradient on that input.

^

This is a bit strict, but we didn't want to list different ways something could be isomorphic to the gradient.
Basic Facts Beanbag
by Screwtape — 2023-03-09T00:05:12.096Z

Summary: Take a beanbag. Pick a topic of simple information, such as the one digit multiplication tables or states and capitals or Scott’s Basic Numbers To Have Intuitions About Other Things. Toss the beanbag to someone else, asking a question as you do so. They catch it and answer as fast as they can before passing it to someone else the same way.

Tags: Repeatable, Medium

Purpose: Intended to train quick, basic information. This can be used as a group memory refresher.

Materials: No materials are required. However, a beanbag or similar light and soft object is recommended. I also suggest having printouts of the facts you want to memorize, though it’s fine to have it on your phone instead.

Announcement Text: There are a few basic facts everyone should know. One category is things that are useful to have so you can put other numbers in context. Scott Alexander’s A Few Basic Numbers To Have Intuitions About Other Things suggests sixteen facts which he uses to grapple with the world. We’re going to come together and learn some basics by tossing a beanbag at each other for fun.

Think of it like a rolling trivia game, except you can’t lose and we’re all working together to try and have everyone comfortable with the answers.

Description: To prepare, take a look around your meeting space and check that there’s nothing fragile or easily breakable around. This is not an activity to run in a busy restaurant. Make sure at least one person has The List of facts in front of them, though everyone doesn't all need their own copy. It's sporting to make sure everyone's read The List at least once, but if they haven't they'll learn soon enough.

Hold up The Beanbag and explain the following rules to people.

0. The Count starts at zero.

1. Whoever is holding The Beanbag is going to ask a question from The List. “Two plus two is-”

2. They are then going to toss The Beanbag to someone else, who will then catch the bean bag and try to correctly answer the question as fast as they can. “-Four!”

2. If they are correct, add one to The Count. If they are wrong, The Count resets to zero.

3. Everyone else chants the correct answer, e.g. “Two plus two is four!” Then Everyone chants The Count together.

4. Start from step 1.

5. If someone calls Halt, whoever has The Beanbag hands (not throws) it to the organizer.

Halt can be called for several reasons, but the most important reason ...