trentmkelly/LessWrong-43k huggingface dataset:
A stylized dialogue on John Wentworth's claims about markets and optimization
by So8res — 2023-03-25T22:32:53.216Z

(This is a stylized version of a real conversation, where the first part happened as part of a public debate between John Wentworth and Eliezer Yudkowsky, and the second part happened between John and me over the following morning. The below is combined, stylized, and written in my own voice throughout. The specific concrete examples in John's part of the dialog were produced by me. It's over a year old. Sorry for the lag.)

(As to whether John agrees with this dialog, he said "there was not any point at which I thought my views were importantly misrepresented" when I asked him for comment.)

J: It seems to me that the field of alignment doesn't understand the most basic theory of agents, and is missing obvious insights when it comes to modeling the sorts of systems they purport to study.

N: Do tell. (I'm personally sympathetic to claims of the form "none of you idiots have any idea wtf you're doing", and am quite open to the hypothesis that I've been an idiot in this regard.)

J: Consider the coherence theorems that say that if you can't pump resources out of a system, then it's acting agent-like.

N: I'd qualify "agent-like with respect to you", if I used the word 'agent' at all (which I mostly wouldn't), and would caveat that there are a few additional subtleties, but sure.

J: Some of those subtleties are important! In particular: there's a gap between systems that you can't pump resources out of, and systems that have a utility function. The bridge across that gap is an additional assumption that the system won't pass up certain gains (in a specific sense).

Roughly: if you won't accept 1 pepper for 1 mushroom, then you should accept 2 mushrooms for 1 pepper, because a system that accepts both of those trades winds up with strictly more resources than a system that rejects both (by 1 mushroom), and you should be able to do at least that well.

N: I agree.

J: But some of the epistemically efficient systems around us violate this property.

For instance, consider a market for (at least) two goods: peppers and mushrooms; with (at least) two participants: Alice and Bob. Suppose Alice's utility is UA(p,m):=log10(p)+log100(m) (where p and m are the quantities of peppers and mushrooms owned by Alice, respectively), and Bob's utility is UB(p,m):=log100(p)+log10(m) (where p and m are the quantities of peppers and m...
Reproject on Cropping
by jefftk — 2023-03-25T21:50:01.567Z

Sometimes you'll come across a picture that looks all skewed, usually
where someone cropped a small piece out of the corner of something
larger:

Projecting a 3D scene to make a 2D picture unavoidably introduces
distortion, worse the farther you are from the center, so if you crop
to just the corner of a picture you're getting a lot of distortion.

It would be possible to make this better: your phone knows the
distortions its lens makes, and every time you crop something it could
automatically reproject the image. I can approximate correction with a
vertical shear:

Followed by a horizontal scale:

In writing this up, however, I realized that my phone's (Pixel 5a)
camera app takes a different approach. Instead of waiting for you to
crop an image, it automatically fixes faces itself. The pictures
above were taken with Open Camera, which doesn't have these smarts;
here's what I get with the default camera app:

It made the fix initially, not on cropping; the full-size image also
minimizes distortion around my face:

Since this is only applied to my face, though, and not the rest of my
body, it does give a slightly uncanny effect. Compare to the
full-size version of the Open Camera image:

You can see what the Pixel camera app did around my face by looking at
the line of the bookcase:

In real life and in a raw photo that line would be straight:

And in my manually corrected version from above the line is straight:

But because the Pixel camera is doing its correction on the whole
image, instead of waiting for you to crop, it isn't able to preserve
all the straight lines in the original image.

I think the Pixel edit is probably still worth doing, since we care a
lot about faces and they do look more natural projected as if you were
looking straight at them. Reprojecting the whole image to minimize
distortion when you crop would also be a pretty nifty feature, and
doesn't introduce the kind of distortions that fixing one part of a
larger image does.
Sam Altman on GPT-4, ChatGPT, and the Future of AI | Lex Fridman Podcast #367
by Gabe M — 2023-03-25T19:08:55.249Z
Link: https://www.youtube.com/watch?v=L_Guz73e6fw

Lex Fridman just released a podcast episode with Sam Altman, CEO of OpenAI. In my opinion, there wasn't too much new here that hasn't been said in other recent interviews. However, here are some scattered notes on parts I found interesting from an AI safety lens:

AI risk https://youtu.be/L_Guz73e6fw?t=3266

Lex asks Sama to steelman Eliezer Yudkowsky's views
Sama said there's some chance of no hope, but the only way he knows how to fix things is to keep iterating and eliminating the "1-shot-to-get-it-right" cases. He does like one of Eliezer's posts that discusses his reasons why he thinks alignment is hard [I believe this is in reference to AGI Ruin: A List of Lethalities].
Lex confirms he will do an interview with Eliezer.
Sama: Now is the time to ramp up technical alignment work.
Lex: What about fast takeoffs?
Sama: I'm not that surprised by GPT-4, was a little surprised by ChatGPT [I think this means this feels slow to him]. I'm in the long-takeoffs, short-timelines quadrant. I'm scared of the short-takeoff scenarios.
Sama has heard of but not seem Ex Machina

On power

https://www.youtube.com/watch?v=L_Guz73e6fw&t=4614s

Sama says it's weird that it will be OOM thousands of people in control of the first AGI .
Acknowledges the AIS people think OAI deploying things fast is bad.
Sama asks how Lex thinks they're doing.
Lex likes the transparency and openly sharing the issues.
Sama: Should we open source GPT-4?
Lex: Knowing people at OAI, no (bc he trusts them,)
Sama: I think people at OAI know the stakes of what we're building. But we're always looking for feedback from smart people.
Lex: How do you take feedback?
Sama: Twitter is unreadable. Mostly from convos like this.

On responsibility https://youtu.be/L_Guz73e6fw?t=6813

Sama: We will have very significant but new and different challenges [with governing/deciding how to steer AI]
Lex: Is it up to GPT or the humans to decrease the amount of hate in the world.
Sama: I think we as OAI have responsibility for the tools we put out in the world, I think the tools can't have responsibility.
Lex: So there could be harm caused by these tools—
Sama: There will be harm caused by these tools. There will be tremendous benefits. But tools do wonderful good and real bad. And we will minimize the bad and maximize the good.

Jailbreaking https://youtu.be/L_Guz73e6fw?t=6939

Lex: How do you prevent jailbreaking?
Sama: It kinda sucks being on the si

...
$500 Bounty/Contest: Explain Infra-Bayes In The Language Of Game Theory
by johnswentworth — 2023-03-25T17:29:51.498Z

Here's my current best guess at how Infra-Bayes works:

We want to get worst-case guarantees for an agent using a Bayesian-like framework.
So, let our agent be a Bayesian which models the environment as containing an adversary which chooses worst-case values for any of the things over which we want worst-case guarantees.
That's just a standard two-player zero-sum game between the agent and the adversary, so we can import all the nice intuitive stuff from game theory.
... but instead of that, we're going to express everything in the unnecessarily-abstract language of measure theory and convex sets, and rederive a bunch of game theory without mentioning that that's what we're doing.

This bounty is for someone to write an intuitively-accessible infrabayes explainer in game theoretic language, and explain how the game-theoretic concepts relate to the concepts in existing presentations of infra-bayes. In short: provide a translation.

Here's a sample of the sort of thing I have in mind:

Conceptually, an infrabayesian agent is just an ordinary Bayesian game-theoretic agent, which models itself/its environment as a standard two-player zero-sum game.

In the existing presentations of infra-bayes, the two-player game is only given implicitly. The agent's strategy π solves the problem:

maxπmine∈BEπ⋅e[U]

In game-theoretic terms, the "max" represents the agent's decision, while the "min" represents the adversary's.

Much of the mathematical tractability stems from the fact that B is a convex set of environments (i.e. functions from policy π to probability distributions). In game-theoretic terms, the adversary's choice of strategy determines which "environment" the agent faces, and the adversary can choose from any option in B. Convexity of B follows from the adversary's ability to use mixed strategies: because the adversary can take a randomized mix of any two strategies available to it, the adversary can make the agent face any convex combination of (policy -> distribution) functions in B. Thus, B is closed under convex combinations; it's a convex set.

I'd like a writeup along roughly these conceptual lines which covers as much as possible of the major high-level definitions and results in infra-bayes to date. On the other hand, I give approximately-zero shits about all the measure theory; just state the relevant high-leve...
Aligned AI as a wrapper around an LLM
by cousin_it — 2023-03-25T15:58:41.361Z

In my previous post Are extrapolation-based AIs alignable? I argued that an AI trained only to extrapolate some dataset (like an LLM) can't really be aligned, because it wouldn't know what information can be shared when and with whom. So to be used for good, it needs to be in the hands of a good operator.

That suggests the idea that the "operator" of an LLM should be another, smaller AI wrapped around it, trained for alignment. It would take care of all interactions with the world, and decide when and how to call the internal LLM, thus delegating most of the intelligence work to it.

Q1: In this approach, do we still need to finetune the LLM for alignment?

A: Hopefully not. We would train it only for extrapolation, and train the wrapper AI for alignment.

Q2: How would we train the wrapper?

A: I don't know. For the moment, handwave it with "the wrapper is smaller, and its interactions with the LLM are text-based, so training it for alignment should be simpler than training a big opaque AI for both intelligence and alignment at once". But it's very fuzzy to me.

Q3: If the LLM+wrapper combination is meant to be aligned, and the LLM isn't aligned on its own, wouldn't the wrapper need to know everything about human values?

A: Hopefully not, because information about human values can be coaxed out of the LLM (maybe by using magic words like "good", "Bertrand Russell", "CEV" and so on) and I'd expect the wrapper to learn to do just that.

Q4: Wouldn't the wrapper become a powerful AI of its own?

A: Again, hopefully not. My hypothesis is that its intelligence growth will be "stunted" by the availability of the LLM.

Q5: Wouldn't the wrapper be vulnerable to takeover by a mesa-optimizer in the LLM?

A: Yeah. I don't know how real that danger is. We probably need to see such mesa-optimizers in the lab, so we can train the wrapper to avoid invoking them.

Anyway, I understand that putting an alignment proposal out there is kinda sticking my head out. It's very possible that my whole idea is fatally incomplete or unworkable, like the examples Nate described. So please feel free to poke holes in it.
Good News, Everyone!
by jbash — 2023-03-25T13:48:22.499Z

As somebody who's been watching AI notkilleveryoneism for a very long time, but is sitting at a bit of a remove from the action, I think I may be able to "see the elephant" better than some people on the inside.

I actually believe I see the big players converging toward something of an unrecognized, perhaps unconscious consensus about how to approach the problem. This really came together in my mind when I saw OpenAI's plugin system for ChatGPT.

I thought I'd summarize what I think are the major points. They're not all universal; obviously some of them are more established than others.

Because AI misbehavior is likely to come from complicated, emergent sources, any attempt to "design it out" is likely to fail.

Avoid this trap by generating your AI in an automated way using the most opaque, uninterpretable architecture you can devise. If you happen on something that seems to work, don't ask why; just scale it up.

Overcomplicated criteria for "good" and "bad" behavior will lead to errors in both specification and implementation.

Avoid this by identifying concepts like "safety" and "alignment" with easily measurable behaviors. Examples:

Not saying anything that offends anybody

Not unnerving people

Not handing out widely and easily available factual information from a predefined list of types that could possibly be misused.

Resist the danger of more complicated views. If you do believe you'll have to accept more complication in the future, avoid acting on that for as long as possible.

In keeping with the strategy of avoiding errors by not manually trying to define the intrinsic behavior of a complex system, enforce these safety and alignment criteria primarily by bashing on the nearly complete system from the outside until you no longer observe very much of the undesired behavior.

Trust the system to implement this adjustment by an appropriate modification to its internal strategies. (LLM post-tuning with RLxF).

As a general rule, build very agenty systems that plan and adapt to various environments. Have them dynamically discover their goals (DeepMind). If you didn't build an agenty enough system at the beginning, do whatever you can to graft in agenty behavior after the fact (OpenAI).

Make sure your system is crafty enough to avoid being suborned by humans. Teach it to win against them at games of persuasion and deception (Facebook).

Everybody knows t

...
ChatGPT Plugins - The Beginning of the End
by Bary Levy — 2023-03-25T11:45:32.877Z

This is going to be a short post. OpenAI recently announced ChatGPT Plugins which will allow any developer to connect ChatGPT to external APIs and potentially real-world actions. Even if GPT-4 is not yet capable of destroying civilization, we are entering a world where LLMs are heavily integrated into everything we build, and take actions on their own. Any outage will be economically unacceptable, and there is no way back. It could take only the next few iterations of the language model, and the right feedback loop through some combination of plugins, to yield a catastrophe-capable agent.

I hope us rationalists still have an emotional part, so let me appeal to it. Do not give up. This post isn't meant to discourage. I don't know what we are going to do in this new world, but I'm ready for the fight.
AI Capabilities vs. AI Products
by Darmani — 2023-03-25T01:14:51.867Z

It's a great time to start an AI company. You can create a product in days that would have been unbelievable just two years ago. Thanks to AI, there will soon be a one-person billion-dollar company.

It's a terrible time to start an AI company. The singularity is fast approaching, and any talent that enters the field should instead go work in AI alignment or policy. Living a monastic lifestyle is better than being dead.
But wait! My plan is to just build software that takes helps people count calories by taking pictures of their meals. How does counting calories help bring about the end of the world?

In the past, any company worthy of being called an AI company would be training models to do things AI had never done before. But now it just means using the same pre-built AI models, programming it in English. Calling a calorie-counting app an "AI company" and worrying about its affect on AI timelines sounds a lot like worrying about Korean car manufacturers becoming dominant because there are an increasing number of Korean "car companies" that deliver cookies.
We need to distinguish between AI capabilities companies, which actually advance the state of the art in AI, vs. AI product companies, which merely use it.
So, my two questions:

To what extent do AI product companies increase AI capabilities?
Is it unethical for someone who believes in a high probability of AI doom to found an AI product company?

Simple economics does tell us that, all else being equal, the more AI product companies there are, the better AI capabilities will be. But, at an individual level, should someone considering starting an AI product company shy away from it?
I don't know. Here are some arguments for and against:

Building AI Products is Unethical

Argument: You are increasing AI capabilities by paying money to AI vendors, who will use it to increase capabilities.
Counterargument: Many believe LLMs will become commoditized. Then there will be so many vendors that none will make an economic profit. There are likely to soon be competitive leaked or open source LLMs. You will be able to run these yourself with very little value flowing into AI capabilities creators.
Argument: The barrier to entry is so low that every vertical will face stiff competition. One dimension people will compete on is the power of their foundation models. Even if you aren't paying OpenAI to create a GPT6 which co

...
Nudging Polarization
by jefftk — 2023-03-24T23:50:01.476Z

In a polarized political environment like the US, ideas that start out
neutral often end up aligned with one side or the other. In cases
where there's an important idea or policy that's currently neutral,
and multiple potential implementations that are also neutral, it would
be much better to get polarization around which implementation to
choose than on the core idea. Is there anything we can do to make
this more likely?

Let's look at an example where this didn't happen: covid vaccination
in the US ended mostly liberal-aligned. This was pretty unfortunate:
the vaccines are very effective against death, and a lot of people
died essentially because they had the bad luck to be part of a
constituency that ended up opposed to them. This could have gone the
other way: Operation Warp Speed came very close to getting out a
vaccine before the 2020 election, there was a lot of talk among
liberals about how they didn't trust a rushed Trump vaccine. If
vaccination had ended up conservative-aligned instead, though, we'd
have had the same downsides in the other direction; not an
improvement.

But what if somehow we'd ended up with the mRNA vaccines (new,
progress) as liberal-aligned and the adenovirus ones (traditional,
reliable) as conservative-aligned? With vaccines for both sides of
the political spectrum we'd likely have seen a lot more adoption and
fewer deaths.

Or consider germicidal UV-C light, which is potentially valuable in
reducing risk from future pandemics because it can purify air without
noisy fans. There are two main approaches:

Upper room: shine it well above people's heads. Since it's not
hitting people it's ok to use frequencies and levels that would be bad
if they. This is the traditional approach, which pre-covid was mostly
only still used in special-purpose medical settings like TB
wards. Sometimes called "254" because that's the peak frequency
low-pressure mercury lights produce, though if we were deploying this
widely we'd probably use LEDs around 265nm.

Whole room: shine it down from the ceiling. You can't do this
with 254nm, but with higher frequency light like 222nm (from KrCl) it should
be safe to shine on people. Needs more research, but very
promising.

It would be unfortunate if UV-C in general ended up politically
aligned, where a large portion of the country wouldn't use it. But if,
say, upper-room e...
"Slightly Evil" AI Apps
by intellectronica — 2023-03-24T22:52:45.516Z
Link: https://intellectronica.net/slightly-evil-ai-apps

The term "Slightly Evil", coined by internet guru Venkatesh Rao, refers to actions and choices that are advantageous from a game-theoretical perspective, are not inherentlyevil (that is, not unambiguously evil in the relevant contexts), but are outside the norm as judged from a naive social perspective.

Many of the AI apps that will be emerging in the coming days, weeks, and months and prove unexpectedly successful are likely to fall into this category, at least once all opportunities for straightforward adaptation of legacy services to a chat format will be exhausted.

The Moralising Big Media (MBM - for example the sort of mass publication dailies hailing from New York, Washington, and Manchester) will approach these with a characteristically low threshold and fail to distinguish between the "slightly evil" and the unambiguously, inherently, dangerously evil. And that leaves you and me, dear reader, to exercise our critical thinking muscles, if we are to make sense of this exciting new world.

Let's try a few (obvious) examples:

Communiclone

AI agent capable of communicating with others using text, audio, and video in a manner that is in both form and content indistinguishable to real flesh-and-bones you.

MBM Panic: Initially intended for simple tasks like reserving a table at the restaurant or answering unsolicited phone calls, it is now used by terrible people like a self-centred woman who lets it communicate with her insufferable, aging mother-in-law and an absent husband and father using it to be a pretend workaholic to his family while spending his evenings down at the pub.

Not inherently evil because: the technology is good enough so that the other side never notices; Billions of people around the world will be leading a happier and more productive life now that they don't have to talk to people they never really wanted to talk to.

Mechanical Wingman

Takes the soul-sucking drudgery out of online dating by automating all digital interactions end-to-end from the moment you sign up and until you are ready to go on a real life date, including picking and retouching your best photos and strategically swiping in the optimal direction.

MBM Panic: Algorithms take the romance out of dating. Innocent women and men exploited to go on dates with people they never even talked to.

Not inherently evil because: online dating really is soul-sucking drudgery. All anyone ever...
Hutter-Prize for Prompts
by rokosbasilisk — 2023-03-24T21:26:41.810Z

The aim of the Hutter Prize is to compress the first 1GB of Wikipedia to the smallest possible size. From the AIXI standpoint, compression is equal to AI, and if we can compress this to the ideal size (75MB according to Shannon's lower estimate), then the compression algorithm is equivalent to AIXI.

However, all the winning solutions so far are based on arithmetic encoding, context mixing. These solutions hold little relevance in mainstream AGI research.

Current day LLMs are really powerful models of our world.And it is highly possible that most LLMs are trained on data that consist of the first 1GB of Wikipedia. Therefore, with appropriate prompting, it should be possible to extract most or all of this data from the LLMs.

I am looking for ideas/papers which would help me validate whether it is possible to extract pieces of wikipedia using prompts with any publicly available LLMs.

update: i have recently learnt regarding gpt-4's compression abilities via prompting, i am very keen in testing this using this method. If anyone is willing to work with me (as i do not have access to gpt-4) it would be great.