trentmkelly/LessWrong-43k huggingface dataset:
Infrafunctions Proofs
by Diffractor — 2023-04-27T19:25:44.903Z

Proposition 1:

Given some compact metric space of options X, if U:X→R is a bounded function, {μ|∀ν∈ΔX:μ(U)≥ν(U)}=Δ{x|∀y∈X:U(x)≥U(y)}

PROOF: To do this, we must show that anything in the first set is in the second, and vice-versa. So, assume μ is in the first set. Then for any ν, ν(U)≤μ(U). However, if μ is not in the second set, then it has probability-mass on some points x where there's a y s.t. U(y)>U(x). Moving that new probability mass to points of higher utility, we'd construct a new probability distribution ν s.t. ν(U)>μ(U), which is impossible. So μ must be in the second set.

In the reverse direction, any probability distribution μ supported over the maximum of U (let x be an arbitrary point in it) has μ(U)=U(x). Thus, for any ν, it's supported entirely over points with equal or lesser utility than this maximum value, so μ(U)≥ν(U). Both subset inclusion directions have been proved, so the equality holds.

Theorem 1: Fundamental Theorem of Infrafunctions

There is a bijection between concave upper-semicontinuous functions of type ΔX→R∪{−∞}, and closed convex upper-complete sets of continuous functions X→R.

Proof: For the reverse direction, going from sets to functions, letting ^F denote the set and F denote the function, we have that
F(μ):=inff∈^Fμ(f)
With this definition being made, let's verify that the resulting function fulfills the indicated property. For verifying concavity, we have
F(pμ+(1−p)ν)=inff∈^F(pμ+(1−p)ν)(f)≥pinff∈^Fμ(f)+(1−p)inff∈^Fν(f)=pF(μ)+(1−p)F(ν)
For verifying upper-semicontinuity, let μn limit to μ.
Now, for any particular function f∗, due to μn limiting to μ, we have
limsupn→∞μn(f∗)=μ(f∗)
So, for any f∗ where μ(f∗) is an (arbitrarily close) approximation to inff∈^Fμ(f), we can go
F(μ)=inff∈^Fμ(f)≃μ(f∗)=limsupn→∞μn(f∗)≥limsupn→∞inff∈^Fμn(f)=limsupn→∞F(μn)
And so upper-semicontinuity is established, that
F(μ)≥limsupn→∞F(μn)
For the forwards direction, going from functions to sets, letting F denote the function, the corresponding set of functions is
{f|∀μ∈ΔX:μ(f)≥F(μ)}
We must show that the result is closed, convex, and upper-complete.

For closure, if fm limits to f, and all the fm are in the set (so μ(fm)≥F(μ) for all m and μ) and μ is arbitrary, then due to fm limiting to f, we have that μ(f)=limm→∞μ(fm)≥F(μ) and this argument works for all μ, so f is in the indicated set.

For convexity, our job is to show that if f and g...
Infrafunctions and Robust Optimization
by Diffractor — 2023-04-27T19:25:11.662Z

Proofs are in this link

This will be a fairly important post. Not one of those obscure result-packed posts, but something a bit more fundamental that I hope to refer back to many times in the future. It's at least worth your time to read this first section up to its last paragraph.

There are quite a few places where randomization would help in designing an agent. Maybe we want to find an interpolation between an agent picking the best result, and an agent mimicking the distribution over what a human would do. Maybe we want the agent to do some random exploration in an environment. Maybe we want an agent to randomize amongst promising plans instead of committing fully to the plan it thinks is the best.

However, all of these run into the standard objection that any behavior like this, where a randomized action is the best thing to do, is unstable as the agent gets smarter and has the ability to rewrite itself. If an agent is randomizing to sometimes take actions that aren' t optimal according to its utility function, then there will be an incentive for the agent to self-modify to eliminate its randomization into those suboptimal actions.

The formalization of this is the following proposition.

Proposition 1: Given some compact metric space of options X, if U:X→R is a bounded function, {μ|∀ν∈ΔX:μ(U)≥ν(U)}=Δ{x|∀y∈X:U(x)≥U(y)}

Intuitively, what this is saying is that the only possible way for a mixture of options to be an optimal move is if each component option is an optimal move. So, utility functions can only give you randomization behavior if the randomization is between optimal actions. The set of such will typically only contain a single point. And so, in general, for any utility function at all, an agent using it will experience a convergent pressure towards deterministic decision-making.

Every single clever alignment trick involving an agent behaving randomly or sampling from a distribution is thereby guaranteed to fail, as it's not stable under reflection as the agent gets smarter, for anything worthy of being called an agent (in the sense that it has an implicit utility function and acts to achieve it).

The rest of this post will be about how the above sentence is false. There's a mathematically principled, reflectively stable, way an agent can be, where randomization behavior persists. No matter how smart it gets, it won't want to remove its randomization behav...
What are the limits of superintelligence?
by rainy — 2023-04-27T18:29:32.081Z

This post is mostly concerned with a superintelligent AI performing recursive self-improvement, this analysis is done to help make sense of the take off speed of such an operation.

Plausibility and Limits

Before considering upper limits, it may be worth considering whether general superintelligence is possible at all. It has been suggested that the idea of recursive self-improvement is similar to the infamous concept of a "perpetual motion machine". We know that a p.m.m. is impossible because it violates thermodynamics. Is there an analogous proof or argument that shows recursive self-improvement impossible? Some good places to start looking for hard limits on superintelligence are mathematics, computability and physics. It's also useful to think about this in terms of biology:

Biology

One example of incremental intelligence being improved upon is evolution: evolution has produced humans from apes. It is possible to simulate evolution in a computer, but evolution takes a very long time. This gives at least a kind of weak existence argument that improving intelligence is possible - and in fact doing so did not require any intelligence (evolution is "dumb"). If a blind process like evolution can do it, it seems likely that a specific problem solving algorithm aimed at doing so could pull it off more efficiently.

Is it possible for something more intelligent than humans to exist? I don't think there is any reason why humans would be at the ultimate limit of intelligence. Computers are far superior at some extremely specific tasks. It's realistic to imagine something basically human level but with computational excellence at a long list of tasks like arithmetic, exact recall of TBs of data, etc. You can also imagine that if it is possible to simulate a human mind and body in a computer, then running one at 64x speed up could be considered a super intelligence, or if that is intractable then running 65535 of them at a slightly slower speed in parallel may also be.

Mathematics

Do facts like Godel's incompleteness theorem or Alan Turing's Halting Problem or Rice's theorem tell us that an intelligence algorithm cannot study and improve itself? I would argue that these theorems do not imply that.

The halting problem is telling us that there isn't a perfect algorithm that can decide every single possible instance of halting, for all inputs. This is a very strong restriction. ...
A Proposal for AI Alignment: Using Directly Opposing Models
by Arne B — 2023-04-27T18:05:44.229Z

The following text proposes a potential solution to the AI alignment problem. I am sharing it here as I have not come across any major issues with the proposed approach, such as those faced by the AI Stop button or the unworkable schemes outlined in the AGI Ruin: A List of Lethalities article. However I am still sceptical about this approach, so if you do have any problems or critiques that I have not addressed under the problems or assumptions sections, please feel free to share them in the comments below.

The Idea

The proposed solution involves using different AIs to control the model. While this idea has been suggested previously, it is not safe on its own as the AI's could potentially cooperate to achieve a shared objective. To address this issue, the following system has been designed to make cooperation impossible by assigning directly opposite goals to the AIs. The system utilizes reinforcement learning to ensure that the AIs work towards opposing objectives, preventing them from colluding. After explaining the idea, I have outlined the assumptions that were made and the issues associated with this approach. The system is comprised of the following components:

Agent

The Agent refers to the model that requires alignment, such as GPT-4.

Police

The Police is a separate model from the Agent that receives the Agent's inputs and outputs before the outputs are used. Its role is to ensure that the Agent does not engage in any inappropriate behavior. If the Police detects misconduct, such as the Agent providing explanations for illegal activities, it can use reinforcement learning to punish the Agent. The Police is rewarded when it successfully punishes the Agent. However, using only the Police model is not advantageous because it will lead to the Agent being punished continuously, rendering it ineffective. To prevent this, the Police model is paired with an opposing model, the Defendant.

Defendant

The Defendant model is also fed the Agent's input and output. However, its task is to reward the Agent for behaviour that isn't bad. When rewarding the Agent, the Defendant also gets rewarded.

The Agent cannot be punished and rewarded at the same time, so the Police and Defendant must give the same answer when asked whether the Agent behaved positively or negatively. When they do not, a fourth Entity, the Judge, is consulted to make the final decision.

Judge

The Judge assesses...
My views on “doom”
by paulfchristiano — 2023-04-27T17:50:01.415Z

I’m often asked: “what’s the probability of a really bad outcome from AI?”

There are many different versions of that question with different answers. In this post I’ll try to answer a bunch of versions of this question all in one place.

Two distinctions

Two distinctions often lead to confusion about what I believe:

One distinction is between dying (“extinction risk”) and having a bad future (“existential risk”). I think there’s a good chance of bad futures without extinction, e.g. that AI systems take over but don’t kill everyone.
An important subcategory of “bad future” is “AI takeover:” an outcome where the world is governed by AI systems, and we weren’t able to build AI systems who share our values or care a lot about helping us. This need not result in humans dying, and it may not even be an objectively terrible future. But it does mean that humanity gave up control over its destiny, and I think in expectation it’s pretty bad.
A second distinction is between dying now and dying later. I think that there’s a good chance that we don’t die from AI, but that AI and other technologies greatly accelerate the rate of change in the world and so something else kills us shortly later. I wouldn’t call this “from AI” but I do think it happens soon in calendar time and I’m not sure the distinction is comforting to most people.

Other caveats

I’ll give my beliefs in terms of probabilities, but these really are just best guesses — the point of numbers is to quantify and communicate what I believe, not to claim I have some kind of calibrated model that spits out these numbers.

Only one of these guesses is even really related to my day job (the 15% probability that AI systems built by humans will take over). For the other questions I’m just a person who’s thought about it a bit in passing. I wouldn’t recommend deferring to the 15%, but definitely wouldn’t recommend deferring to anything else.

A final source of confusion is that I give different numbers on different days. Sometimes that’s because I’ve considered new evidence, but normally it’s just because these numbers are just an imprecise quantification of my belief that changes from day to day. One day I might say 50%, the next I might say 66%, the next I might say 33%.

I’m giving percentages but you should treat these numbers as having 0.5 significant figures.

My best guesses

Probabilit...
Quote quiz: “drifting into dependence”
by jasoncrawford — 2023-04-27T15:13:12.141Z
Link: https://rootsofprogress.org/quote-quiz-drifting-into-dependence

Quote quiz: who said this? (No fair looking it up). I have modified the original quotation slightly, by making a handful of word substitutions to bring it up to date:

It might be argued that the human race would never be foolish enough to hand over all power to AI. But we are suggesting neither that the human race would voluntarily turn power over to AI nor that AI would willfully seize power. What we do suggest is that the human race might easily permit itself to drift into a position of such dependence on AI that it would have no practical choice but to accept all of the AI’s decisions. As society and the problems that face it become more and more complex and as AI becomes more and more intelligent, people will let AI make more and more of their decisions for them, simply because AI-made decisions will bring better results than man-made ones. Eventually a stage may be reached at which the decisions necessary to keep the system running will be so complex that human beings will be incapable of making them intelligently. At that stage the AI will be in effective control. People won’t be able to just turn the AI off, because they will be so dependent on it that turning it off would amount to suicide.

I’ll post the answer, and the unedited original quotation, next week.

UPDATE: Here's the answer.
Second-Level Empiricism: Reframing the Two-Child Puzzle
by Richard Henage — 2023-04-27T15:04:22.842Z

Empiricism is the belief that knowledge derives from sensory experience, or, by some definitions, experiment. However, empiricism itself can show that it is not the best tool for a given scenario. This article provides an example of when this happened to me.

The Two-Child Puzzle is a logic problem. The version I first heard goes like this:

I know my co-worker has exactly two kids that each have an independent 50/50 chance of being a boy or a girl. I ask her if at least one of her kids is a boy. She tells me that is correct. The question is: From my perspective, what is the probability that the other one is a boy as well?

The answer, I immediately supposed, was 1/3, while the trick was that most readers would guess 1/2. I posed the question to a friend and asked him what he thought. He thought it was clearly 1/2, and said multiple times throughout our ensuing discussion that he was very confident. I tried a variety of tactics to persuade him, initially using pure logic. However, the complexity of language and my inexperience as a teacher made us run in circles, as he would continue to state something like "but what we know about one of the kids can't change the probability of the other" or "if she brought a boy to work one day, instead of telling you she had a boy, you would think that the other kid had a 1/2 chance of being a boy instead of 1/3".

I attempted to use more empirical tactics. First, I used coins to demonstrate the puzzle. They happened to come up with the "girl" answer the first three times, which is more likely if the probability is 1/3 of being a boy than 1/2 of being a boy, yet the method seemed unreliable and my friend was highly skeptical.

Then, I created a spreadsheet with 1000 rows using random values, showing different averages like "both girls", "both boys", "one or more boys", "kid 1 is a boy", and "kid 2 is a boy". I showed that when all results of "both girls" were factored out, the spreadsheet was repeatedly displaying a 1/3 chance of "both boys". Yet my friend continued to state that one kid being a boy was distinct from kid 1 being a boy, and that therefore the spreadsheet was irrelevant (The spreadsheet was still somewhat useful, as I learned that the chance for kid 1 being a boy was 2/3).

My friend called someone who he described as smart and math-oriented and we explained the puzzle. The math-oriented friend ...
Interview with Paul Christiano: How We Prevent the AI’s from Killing us
by Dalmert — 2023-04-27T14:39:49.571Z
Link: https://www.youtube.com/watch?v=GyFkWb903aU

0:00 Intro
9:20 Percentage Likelihood of Death by AI
11:24 Timing
19:15 Chimps to Human Jump
21:55 Thoughts on ChatGPT
27:51 LLMs & AGI
32:49 Time to React?
38:29 AI Takeover
41:51 AI Agency
49:35 Loopholes
51:14 Training AIs to Be Honest
58:00 Psychology
59:36 How Solvable Is the AI Alignment Problem?
1:03:48 The Technical Solutions (Scalable Oversight)
1:16:14 Training AIs to be Bad?!
1:18:22 More Solutions
1:21:36 Stabby AIs
1:26:03 Public vs. Private (Lab) AIs
1:28:31 Inside Neural Nets
1:32:11 4th Solution
1:35:00 Manpower & Funding
1:38:15 Pause AI?
1:43:29 Resources & Education on AI Safety
1:46:13 Talent
1:49:00 Paul’s Day Job
1:50:15 Nobel Prize
1:52:35 Treating AIs with Respect
1:53:41 Uptopia Scenario
1:55:50 Closing & Disclaimers
AI #9: The Merge and the Million Tokens
by Zvi — 2023-04-27T14:20:01.329Z

There were two big developments this week.

One is that Google merged Google Brain and DeepMind into the new Google DeepMind. DeepMind head and founder Demis Hassabis is in charge of the new division. We will see how this plays out in practice, seems very good for Google.

The other is that a paper claims to have figured out how to extend the context window to one… million… tokens, making the cost per token linear rather than quadratic. If so, and this doesn’t have big other downsides, that’s a huge game. A comment suggests this is more on the 'in mice' level, so perhaps we should not get our hopes up.

Lots of other stuff happened as well, those are the two must-know items.

Split off this week were:

Transcript and Brief Response to Twitter Conversation Between Yann LeCunn and Eliezer Yudkowsky. It is important to preserve such things and put them into readable form. That does not mean any given person must read them.

Notes on Potential Future AI Tax Policy. Target the training runs, if possible.

A Hypothetical Takeover Scenario Twitter Poll. I asked Twitter about a scenario where an ASI attempts a world takeover to kill all humans, but has its abilities sharply restricted to the practical and known (e.g. no foom of any kind, no nanotech, a human-level plan and so on) to see where people were doubtful, where they were confident, what arguments they had and what might be a crux that would change their minds. Mixed results.

Table of Contents

Introduction.

Table of Contents.

Large Language Models Offer Mundane Utility. Have a voice conversation, do an AI Snapchat chat, give the AI the confidence it needs.

Language Models Don’t Offer Mundane Utility. …and it’s gone.

Fun With Image, Audio and Video Generation. Come to Pepperoni Hug Spot.

Did ChatGPT Save Chris Munn Over $100k? Mostly, no.

MedicGPT. Act now, while lack of crippling regulations lasts.

Deepfaketown and Botpocalypse Soon. Worth a few minutes of your time.

Plug-Ins as SuperPrompts. It’s all about the super prompts.

Age of Massive Scale-Based AI Improvements Already Ending? Altman says yes.

It’s Time for the Merge. DeepMind and Google Brain are now Google DeepMind.

Europe’s Ongoing War on the Internet and All Human Activity Potentially Bears Dividends. OpenAI looks to be in quite serious legal trouble in Europe, in ways that are not going to

...
AI doom from an LLM-plateau-ist perspective
by Steven Byrnes — 2023-04-27T13:58:10.973Z

(in the form of an FAQ)

Q: What do you mean, “LLM plateau-ist”?

A: As background, I think it’s obvious that there will eventually be “transformative AI” (TAI) that would radically change the world.[1]

I’m interested in what this TAI will eventually look like algorithmically. Let’s list some possibilities:

A breakdown of possibilities for how future TAI will work. I’ll refer back to this diagram throughout the post.

A “Large Language Model (LLM) plateau-ist” would be defined as someone who thinks that categories (A-B), and usually also (C), will plateau in capabilities before reaching TAI levels.[2] I am an LLM plateau-ist myself.[3]

I’m not going to argue about whether LLM-plateau-ism is right or wrong—that’s outside the scope of this post, and also difficult for me to discuss publicly thanks to infohazard issues.[4] Oh well, we’ll find out one way or the other soon enough.

In the broader AI community, both LLM-plateau-ism and its opposite seem plenty mainstream. Different LLM-plateau-ists have different reasons for holding this belief. I think the two main categories are:

Theoretical—maybe they have theoretical beliefs about what is required for TAI, and they think that LLMs just aren’t built right to do the things that TAI would need to do.
Empirical—maybe they’re not very impressed by the capabilities of current LLMs. Granted, future LLMs will be better than current ones. But maybe they have extrapolated that our planet will run out of data and/or compute before LLMs get all the way up to TAI levels.

Q: If LLMs will plateau, then does that prove that all the worry about AI x-risk is wrong and stupid?

A: No no no, a million times no, and I’m annoyed that this misconception is so rampant in public discourse right now.

(Side note to AI x-risk people: If you have high credence that AI will kill everyone but only medium credence that this AI will involve LLMs, then maybe consider trying harder to get that nuance across in your communications. E.g. Eliezer Yudkowsky is in this category, I think.)

A couple random examples I’ve seen of people failing to distinguish “AI may kill everyone” from “…and that AI will definitely be an LLM”:

Venkatesh Rao’s blog post

“Beyond Hyperanthropomorphism”

goes through an elaborate 7000-word argument that eventually culminates, in the final section, in his assertion that a language model trained on inte

...