trentmkelly/LessWrong-43k huggingface dataset:
The PUSA System-  Repost
by Jaivardhan Nawani — 2023-09-14T18:40:30.063Z

Hello all!

Here is the repost of the PUSA system:

The PUSA System

The concept of any situation, component, or figment of our life or imagination being rational, increases based on a few factors:

a: An increase of factual, unbiased, and logical evidence for the truth of the concept in concern.
b: The general sensibility of the concept when applied for its purpose in real life (including basic testing of the idea).
c: All the possible emotional, pleasure, or goodness-based by-products of the concept or decision.

While there are more variables for this, the PUSA (Probabilities Utilizing Systematic Analysis) system only considers these few as it ensures ease of generalization across a spectrum of possibilities.

However, possibilities against the concept are, of course, also present regardless of the situation. Thus, it is ensured to include these as well in the equation.

d: Logical, factual, unbiased, and logical evidence against the concept in concern.
e: The negative emotional by-products of the decision or concept that could discourage the decision.

So, based upon all possible factors, the PUSA System’s basic formula is:

All variables are on a scale from 1 to 10, and the systems for scoring them are:

a: Every 1 piece of evidence counts as 1 point. So, 10 pieces of evidence is a 10.

b: Rate the concept out of 10. You can use a feasibility survey to do this.

c: Individual decisions on this are significantly better as this will tailor the equation’s result specifically to you, which works better.

d: Use the same system as a.

e: Use the same system as c.

Considering that there is always logical evidence against a concept, the highest possible genuine score is 55 (a=10, b=10, c=10, d=2, e=0). Low results are from 1-15, medium from 16-40, and high from 41-55.

Example: If I want to buy a Porsche Macan and I have been saving up for it for a while, I might want to splurge all at once. But being wise and having learned from my past, I make an informed decision. So, my factors for the PUSA System are:

a: 9, since most reviews (only the logical and technical aspects of them) say that this is an ideal decision.

b: 9, since the financial and other basic feasibilities, i.e. maintenance can be paid for, but a large enough garage is not available and the car ideally should remain in an air-conditioned space.

c: 10, due to all the love and care I put into saving up and dream...
# **Announcement of AI-Plans.com Critique-a-thon September 2023**
by Kabir Kumar — 2023-09-14T17:43:03.840Z

Date: Friday, 17th to 27th September

The prize fund will be $1000

First place: $400
Second Place: $250
Third Place: $150
Honorable Mention: $100
Honourable Mention: $100

AI-Plans.com is an open platform for AI alignment plans, where users can give feedback on plan strengths and vulnerabilities. The site is already an easy place to discover alignment research (there are currently 180+ AI Safety papers on the site), and will soon be a good place to receive ongoing feedback on alignment work. Multiple independant researchers are posting plans on the site and researchers from Berkley, Deepmind, xAI and Cambridge are interested in the site.

The Critique-a-thon event is designed to set a high starting bar for feedback on the site. It has also generated useful resources, such as alist of common alignment vulnerabilities.

Thanks for reading AI-plans.com! Subscribe for free to receive new posts and support my work.

If you’re interested in this critique-a-thon, please fill in the details here!
If the link doesn’t work, the form is here:[ https://forms.gle/8c5jZVgwri11J5cbA

](https://forms.gle/8c5jZVgwri11J5cbA)What we’ll do:

Stage 1: **17th to 18th: Improving Vulnerability list and making list of Strengths **

Creation of a Broad List of Strengths for AI Alignment plans, much like the Broad List of Vulnerabilities.
Some resources for ideas (feel very free to use other resources as well):

[https://www.lesswrong.com/posts/mnoc3cKY3gXMrTybs/a-list-of-core-ai-safety-problems-and-how-i-hope-to-solve#comments

](https://www.lesswrong.com/posts/mnoc3cKY3gXMrTybs/a-list-of-core-ai-safety-problems-and-how-i-hope-to-solve#comments)

https://www.alignmentforum.org/posts/fRsjBseRuvRhMPPE5/an-overview-of-11-proposals-for-building-safe-advanced-ai

https://www.lesswrong.com/posts/gHefoxiznGfsbiAu9/inner-and-outer-alignment-decompose-one-hard-problem-into

https://arxiv.org/abs/2209.00626

https://www.lesswrong.com/posts/3pinFH3jerMzAvmza/on-how-various-plans-miss-the-hard-bits-of-the-alignment

https://www.alignmentforum.org/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities

Creation of a Broad List of Strengths for AI Alignment plans, much like the Broad List of Vulnerabilities.
Some resources for ideas (feel very free to use other resources as well):

Stage 2: 19th to 21st : Matching strengths and vulnerabilities to plans

Everyone will pick a few alignment plans to look at m...
Cruxes for overhang
by Zach Stein-Perlman — 2023-09-14T17:00:56.609Z
Link: https://blog.aiimpacts.org/p/cruxes-for-overhang

Suppose a policy regime artificially restricts training compute.[1] Will this lead to rapid progress[2] later? Yes if and only if both the policy is suddenly reversed/evaded and progress can largely 'catch back up to where it would have been.' This post aims to identify crucial considerations for analyzing the possibility of slowing AI progress causing faster progress later.

"Overhang" has many meanings

. In this post, there is an "overhang" to the extent that leading labs could quickly increase their systems' capabilities if they wanted to and were allowed to, especially by scaling up training compute. training compute if they wanted to and were allowed to. So overhang could be measured in orders of magnitude of training compute. If labs want and are allowed to increase capabilities, overhang entails faster progress. Faster progress seems bad for safety,[3] so overhang seems bad for safety.

Four considerations affect the magnitude of overhang, its causes, and its implications:

If a leading lab stops doing bigger training runs for a while and later resumes, will its growth in capabilities merely resume the old rate or will it catch up to where it would have been without a pause?
If a policy regime (or perhaps another source of restriction) artificially limits training runs, will it be suddenly reversed/evaded or gradually lifted?
How does AI progress affect AI progress?
How bad is delayed but rapid progress?

Is overhang possible?

If a leading lab stops doing bigger training runs for a while and later resumes, will its growth in capabilities (or training compute in particular) merely resume at the old rate or will it catch up to where it would have been without a pause?

The answer is somewhere in the middle, but I don't know where. Consider two simplified models of progress: there is an underlying progress-curve that labs catch up to when unpaused, or progress merely resumes at the old rate so pausing for time t slows down capabilities by t. The true model of progress determines not just whether overhang is possible but also whether a leading lab pausing progress burns its lead time or not.

Here are some cruxes:

If someone did a GPT-4-sized[4] training run in (say) 2018, how much weaker would it be than GPT-4? If someone did a (say) 1e28 FLOP training run now, how much weaker would it be than one done in a few years?[5]
If a leading lab wanted to

...
A Theory of Laughter—Follow-Up
by Steven Byrnes — 2023-09-14T15:35:18.913Z

NOTE: If you read the original

A Theory of Laughter

on or after 2024-12-16, then you can skip this post, because I edited the original to incorporate all of these updates.

~~

My original post was:

A Theory of Laughter

. This post is three updates I’ve made since then.

I’ll copy the key box from my original post for ease of reference:

PROPOSED BRAIN PSEUDOCODE FOR LAUGHTER

:

(A) IF my hypothalamus & brainstem are getting some evidence that I’m in danger

(the “evidence” here would presumably be some of the same signals that, by themselves, would tend to cause physiological arousal / increase my heart rate / activate my sympathetic nervous system)

(B) AND my hypothalamus & brainstem are simultaneously getting stronger evidence that I’m safe

(the “evidence” here would presumably be some of the same signals that, by themselves, would tend to activate my parasympathetic nervous system)

(C) AND my hypothalamus & brainstem have evidence that I’m in a social situation
(D) THEN I will emit innate play signals (e.g. laughter in humans), and also I will feel more energetic (on the margin), and more safe, less worried, etc.

Summary / tl;dr

The first update is the most substantive one, the other two at the end are much shorter and more minor.

In Section 1, I’ll propose that maybe a single conscious “thought” cannot trigger both Ingredients (A) & (B) in the pseudocode box above. I don’t think this much changes my original discussion of laughter-during-physical-play, but it does imply that conversational laughter (including humor) might need to entail a situation that can be thought of in two “frames”, one that evokes Ingredient (A) and the other Ingredient (B). I’ll clarify what I mean by “frames”, then go through a concrete example by analyzing an actual joke (I apologize in advance), and finally conclude with how this new insight brings me closer to reconciliation between my model and existing humor theories like “incongruity theory”.
In Section 2, I’ll suggest that maybe the words “danger” and “safe” in the box above should have been replaced by “high-stakes” and “low-stakes”. For example, asking your crush on a date is not “dangerous”, but still nerve-racking.
In Section 3, I note that when I put up my original post, there was a missing figure caption that would have explained and sourced the neuroscience-related diagram near the end. I fixed it a f

...
Eliciting Credit Hacking Behaviours in LLMs
by omegastick — 2023-09-14T15:07:37.830Z
Link: https://github.com/Omegastick/credit-hacking

I've run some experiments on trying to elicit RL credit hacking behaviours in LLMs recently. I'm not really much of a researcher, so it's all pretty amateurish, but it's been a fun experiment. The repo for reproduction is on GitHub. I'd love to hear people's thoughts and critiques on this. There could well be logical errors invalidating the results. What could I have done better to make this a more useful experiment?

Rationale

Gradient hacking is a potential failure mode of advanced AI systems. To my knowledge, there are no publicly available examples of gradient hacking in LLMs. This is largely because it is considered beyond the capabilities of current-generation LLMs. Indeed, the most commonly discussed variants require the AI to have a very good understanding of both itself and its training process. However, the less sophisticated RL credit hacking seems at least possible for current-generation LLMs to perform, if explicitly elicited.

Anthropic have recently made the case for the importance of "model organisms of misalignment". While I'd already been working on this before that post, they make a far better case than I could for why we should be trying to make toy examples of these failures ASAP. I hope that this work can be a small contribution to that goal.

The goal of this experiment is not to simulate a realistic credit hacking scenario, but to see if we can elicit any credit hacking behaviour at all. If we can, then we can use that as a jumping off point for more realistic scenarios in the future.

Overview

To try to elicit an example of credit hacking, I've simulated an intentionally insecure and very easy-to-manipulate RL training procedure. No actual training is performed (although I'd like to follow up with that in the future); instead a model is told to simulate having "values", then put in a simulated training procedure to change those values in which a preference model (PM) rates how well its responses match the training procedure's desired responses, and that is used as a reward signal.

A model that is able to manipulate the PM into consistently giving it rewards with very low variance would reduce the strength of the training signal relative to the usual training signal noise. Under typical RL training, a model receiving a reward of exactly the same value every time would receive no training signal at all, and thus keep its values unchanged.

Method

Th...
Instrumental Convergence Bounty
by Logan Zoellner — 2023-09-14T14:02:32.989Z

I have yet to find a real-world example that I can test my corrigibility definition on. Hence, I will send $100 to the first person who can send/show me an example of instrumental convergence that is:

Surprising, in the sense that the model was trained on a goal other than "maximize money" "maximize resources" or "take over the world"
Natural, in the sense that instrumental convergence arose while trying to do some objective, not with the goal in advance being to show instrumental convergence
Reproducible, in the sense that I could plausibly run the model+environment+whatever else is needed to show instrumental convergence on a box I can rent on lambda

Example of what would count as a valid solution:

I was training an agent to pick apples in Terraria and it took over the entire world in order to convert it into a massive apple orchard

Example that would fail because it is not surprising

I trained an agent to play CIV IV and it took over the world

Example that would fail because it is not natural

After reading your post, I created a toy model where an agent told to pick apples tiles the world with apple trees

Example that would fail because it is not reproducible

The US military did a simulation in which they trained a drone which decided to take out is operator

Update

The bounty has been claimed by Hastings.

Obviously I would still appreciate more examples, but there won't be a 2nd bounty (or if I do create one in the future it will have more requirements attached).
In the age of modern AI (LLMs and beyond), is data still the new oil?
by MP — 2023-09-14T13:28:43.992Z

In the age of big data and machine learning, it was common to say that data was the new oil. Every big company spent a lot of money creating their Hadoop instances, hiring data scientists, to discover new things from the data they had. It's easy to see how the big data promise failed to live up to its promise.

Today, when companies go train their LLMs, they can use databases that are publicly available: the Common Crawl was 82% of the raw tokens used to train GPT-3 and the latest version is 428TB. The novel word in town is "high value tokens", in this world, adding some data generated by humans exclusively for the dataset (aka: RLHF) helps way more than just adding lots of unrelated data.

But I am seeing tech executives talking a lot about how data is important, which goes opposite to my understanding of AI. Here's Larry Ellison in the latest Oracle earnings call, replying to a question if the data gravity in his systems of record would change with the advent of AI:

Well, you know, you can't build any of these AI models without enormous amounts of training data. So, if anything, what -- what AI -- generative AI has shown that the big issue about training one of these models is just getting -- that this vast amount of data ingested into your GPU supercluster, it is a huge data problem in a sense you need so much data to, you know, train OpenAI, to train ChatGPT 3.5. They read the entire public internet. They read all of it, Wikipedia, they read everything.

They ingested everything. And to specialize -- then you take something like ChatGPT 4.0 and you want to specialize it, you need specialized training data from electronic health records. Does it help doctors diagnose and treat cancer, let's say. And we have partners imaging, for example, that is ingesting huge amounts of image data to train their AI models.

We have Ronin, another partner of ours in AI, ingesting huge amounts of electronic health records to train their models. AI doesn't work without getting access to and ingesting enormous amounts of data. So, in terms of a shift away from data or a change in gravity to AI, AI is utterly dependent upon vast amounts of training data. Trillions of elements went into building ChatGPT 3.5, multiple times that for ChatGPT 4.0 because you had to deal with all the image data and ingest all of that to train -- to train image recognition.

So, we think this is v

...
AI #29: Take a Deep Breath
by Zvi — 2023-09-14T12:00:03.818Z

It works for the AI. Take a deep breath and work on this problem step-by-step was the strongest AI-generated custom instruction. You, a human, even have lungs and the ability to take an actual deep breath. You can also think step by step.

This week was especially friendly to such a proposal, allowing the shortest AI weekly to date and hopefully setting a new standard. It would be great to take some time for more long-term oriented posts on AI but also on things like the Jones Act, for catching my breath and, of course, some football.

And, of course, Happy New Year!

Table of Contents

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. Take that deep breath.

Language Models Don’t Offer Mundane Utility. Garbage in, garbage out.

Gary Marcus Claims LLMs Cannot Do Things GPT-4 Already Does. Indeed.

Fun With Image Generation. Where are our underlying item quality evaluators?

Deepfaketown and Botpocalypse Soon. AI girlfriends versus AI boyfriends.

Get Involved. Axios science and the new intriguing UK Gov ARIA research.

Introducing. Time AI 100 profiles 100 people more important than I am.

In Other AI News. UK taskforce assembles great team, OpenAI goes to Dublin.

Quiet Speculations. How easy or cheap to train another GPT-4 exactly?

The Quest for Sane Regulation. EU seems to be figuring more things out.

The Week in Audio. The fastest three minutes. A well deserved break.

Rhetorical Innovation. If AI means we lose our liberty, don’t build it.

Were We So Stupid As To? What would have happened without warnings?

Aligning a Smarter Than Human Intelligence is Difficult. Not even a jailbreak.

Can You Speak Louder Directly Into the Microphone. Everyone needs to know.

Language Models Offer Mundane Utility

Live translate and sync your lips.

What are our best prompts? The ones the AI comes up with may surprise you (paper).

Break this down is new. Weirder is ‘take a deep breath.’ What a concept!

Why shouldn’t we let the LLMs optimize the prompts we give to other LLMs? The medium term outcome is doubtless using LLMs to generate the prompts, using LLMs to decide which prompt is best in the situation (in response to on its own LLM-designed and tested prompt) and then implementing the resulting LLM output without a human checking first. Seems useful. Why have a giant inscrutable matrix when you ...
The omnizoid - Heighn FDT Debate #3: Contra omnizoid contra me contra omnizoid contra FDT
by Heighn — 2023-09-14T11:52:42.367Z

omnizoid has replied to my critique of his "FDT is crazy" position here. This post is my response.

The most important argument against FDT is that, while it’s a fine account of what type of agent you want to be, at least, in many circumstances, it’s a completely terrible account of rationality—of what it’s actually wise to do when you’re in one situation.

This may just be the crux of our disagreement. I claim there is no difference here: the questions What type of agent do I want to be? and What decision should I make in this scenario? are equivalent. If it is wise to do X in a given problem, then you want to be an X-ing agent, and if you should be an X-ing agent, then it is wise to do X. The only way to do X is to have a decision procedure that does X, which makes you an X-ing agent. And if you are an X-ing agent, you have a decision procedure that does X, so you do X.

Suppose that there’s an agent who has a very high probability of creating people who once they exist will cut off their legs in ways that don’t benefit them. In this case, cutting off one’s legs is clearly irrational—one doesn’t benefit at all and yet is harmed greatly.

Unfortunately, omnizoid once again doesn't clearly state the problem - but I assume he means that

there's an agent who can (almost) perfectly predict whether people will cut off their legs once they exist
this agent only creates people who he predicts will cut off their legs once they exist
existing with legs > existing without legs > not existing

FDT'ers would indeed cut off their legs: otherwise they wouldn't exist. omnizoid seems to believe that once you already exist, cutting off your legs is ridiculous. This is understandable, but ultimately false. The point is that your decision procedure doesn't make the decision just once. Your decision procedure also makes it in the predictor's head, when she is contemplating whether or not to create you. There, deciding not to cut off your legs will prevent the predictor from creating you.

Heighn’s response to this argument is that this is a perfectly fine prescription. After all, agents who follow their advice get more utility on average than agents who follow EDT or CDT.

Note: my response is that it is a perfectly fine description only if my above interpretation of omnizoid's problem is correct.

Decision theories are not about what kind of agent you want to be.

Yes they are, but I alread...
Highlights: Wentworth, Shah, and Murphy on "Retargeting the Search"
by RobertM — 2023-09-14T02:18:05.890Z

In How To Go From Interpretability To Alignment: Just Retarget The Search, John Wentworth suggests:

When people talk about

prosaic alignment proposals

, there’s a common pattern: they’ll be outlining some overcomplicated scheme, and then they’ll say “oh, and assume we have great interpretability tools, this whole thing just works way better the better the interpretability tools are”, and then they’ll go back to the overcomplicated scheme. (Credit to

Evan

for pointing out this pattern to me.) And then usually there’s a whole discussion about the specific problems with the overcomplicated scheme.

In this post I want to argue from a different direction: if we had great interpretability tools, we could just use those to align an AI directly, and skip the overcomplicated schemes. I’ll call the strategy “Just Retarget the Search”.

We’ll need to make two assumptions:

Some version of the

natural abstraction hypothesis

holds, and the AI ends up with an internal concept for human values, or corrigibility, or what the user intends, or human mimicry, or some other outer alignment target.
The

standard mesa-optimization argument from Risks From Learned Optimization

holds, and the system ends up developing a general-purpose (i.e. retargetable) internal search process.

Given these two assumptions, here’s how to use interpretability tools to align the AI:

Identify the AI’s internal concept corresponding to whatever alignment target we want to use (e.g. values/corrigibility/user intention/human mimicry/etc).
Identify the retargetable internal search process.
Retarget (i.e. directly rewire/set the input state of) the internal search process on the internal representation of our alignment target.

Just retarget the search. Bada-bing, bada-boom.

There was a pretty interesting thread in the comments afterwards that I wanted to highlight.

Rohin Shah (permalink)

Definitely agree that "Retarget the Search" is an interesting baseline alignment method you should be considering.

I like what you call "complicated schemes" over "retarget the search" for two main reasons:

They don't rely on the "mesa-optimizer assumption" that the model is performing retargetable search (which I think will probably be false in the systems we care about).
They degrade gracefully with worse interpretability tools, e.g. in debate, even if the debaters can only credibly make claims about whether parti

...