trentmkelly/LessWrong-43k huggingface dataset:
Taboo Wall
by Screwtape — 2023-11-06T03:51:09.968Z

Summary: Rationalist Taboo is the skill of being able to talk about the same concepts using different words. Taboo Wall is open discussion, but as the discussion goes on more words get stricken from everyone’s vocabulary.

Tags: Very Experimental, Medium

Purpose: To practice the ability to communicate what you mean without using specific phrases or concepts.

Materials: A large and easily visible space to write words on. A giant whiteboard works, as does a projector displaying a word document on the wall.

Announcement Text: “Rationalist Taboo” is the skill of explaining the ideas you mean to communicate without the use of specific words. It’s a tool with many uses. Sometimes your conversational partner doesn’t know a particular word, so you want to be able to smoothly switch to vocabulary they are familiar with. Other times you and the people you’re talking to have subtly different definitions or associations with a given word, and so you can clear up some confusion by mutually setting that word aside. (Consider: “If a tree falls in a forest and nobody hears it, does it make a sound?” That’s a potentially difficult question! “If a tree falls in a forest and nobody has an auditory experience of it, does it make vibrations in the air?” That is less difficult!)

Today we’ll be practicing this skill, but mostly we’ll be having conversations about whatever we’re interested in. The trick is, throughout the meetup we’ll be adding words to a “taboo list” that we’re not allowed to say. Please continue your conversation around this slight impediment!

Description: Put one word on the wall, and explain that people are free to talk about whatever they’re interested in but that they are not allowed to use that word. Periodically (I suggest intervals of fifteen minutes, based on a wild hunch) words will be added to the list, and those words are also not allowed. Wave your arms to point out the additions of new words. If anyone notices someone using a word you shouldn’t, they should make a buzzing noise, point out which word was used, and they get a point. (The points do not mean anything.) Don’t worry too much about making sure people know when new words are added; they’ll learn fast enough to keep an eye on the big whiteboard.

Variations: One variation here is the Lightning Talk version. Have people give short talks on a subject, only they’re not allowed to use a bunch of wor...
When and why should you use the Kelly criterion?
by Garrett Baker — 2023-11-05T23:26:38.952Z

This is a dialogue made during the online dialogues party. Phil, River, and I (Garrett) talked about the Kelly criterion. I was under the mistaken impression that it would come up in finite games, even without discount factors. This turned out to be wrong! In such betting games, you always want to bet the maximal amount (assuming linear utility in money).

We then talked about how maybe you could save the criterion without bringing in log utility in money or discount rates in infinite games. The two conclusions were you could either have a finite utility function, or care intrinsically about not being broke/maximizing the probability of getting more money than other agents in your world.

The highlight in my opinion is me changing my mind about the finite games thing after using dynamic programming to try to show Phil to be wrong. The proof we came up with was both surprising and elegant to me.

Possible justifications for Kelly

Garrett Baker

So, I've seen some previous explanations of the kelly criterion on LessWrong, and they seem to fall into 3 clusters:

You do the kelly criterion because you have log-utility in money.
You do the kelly criterion because you have linear utility in money, but you can use current money to gain even more in the future.
You do the kelly criterion because you partially update on the market's position.

philh

Some background here for me is I've previously written https://www.lesswrong.com/posts/XnnfYrqaxqvirpxFX/on-kelly-and-altruism and https://www.lesswrong.com/posts/JAmyTWoukk8xzhE9n/ruining-an-expected-log-money-maximizer, they didn't get much karma and I don't know if that's "there's something wrong with them" or "they didn't get seen much" or what

philh

You do the kelly criterion because you have log-utility in money.

This one I think is complicated; I'd say "you do a thing that turns out equivalent to the kelly criterion, but for simpler reasons than why the kelly criterion is derived"

philh

You do the kelly criterion because you have linear utility in money, but you can use current money to gain even more in the future.

In this case I'd say no, if you actually have linear utility in money you should just bet everything every time in most/all game structures I've seen; this results in behavior that is obviously wrong, but that's because linear utility in money is obviously wrong

philh

You do the kelly criterion because you partially upda

...
On Overhangs and Technological Change
by Roko — 2023-11-05T22:58:51.306Z

Imagine an almost infinite, nearly flat plain of early medieval farming villages populated by humans and their livestock (cows, horses, sheep, etc) politically organized into small duchies and generally peaceful apart from rare skirmishes.

Add a few key technologies like stirrups and compound bows (as well as some social technology - the desire for conquest, maneuver warfare, multiculturalism) and a great khan or warlord can take men and horses and conquer the entire world. The Golden Horde did this to Eurasia in the 1200s.

An overhang in the sense I am using here means a buildup of some resource (like people, horses and land) that builds up far in excess of what some new consuming process needs, and the consuming process proceeds rapidly like a mountaineer falling off an overhanging cliff, as opposed to merely rolling down a steep slope.

The Eurasian Plain pre-1200 was in a "steppe-horde-vulnerable-land Overhang". They didn't know it, but their world was in a metastable state which could rapidly turn into a new, more "energetically favored" state where they had been slaughtered or enslaved by The Mongols.

Before the spread of homo sapiens, the vertebrate land animal biomass was almost entirely not from genus homo. Today, humans and our farm animals comprise something like 90% of it. The pre-homo-sapiens world had a "non-civilized-biomass overhang": there were lots of animals and ecosystems, but they were all pointless (no globally directed utilization of resources, everything was just a localized struggle for survival, so a somewhat coordinated and capable group could just take everything).

Why do these metastable transitions happen? Why didn't the Eurasian Plain just gradually develop horse archers everywhere at once, such that the incumbent groups were not really disrupted? Why don't forests just gradually burn a little bit all over the place, so that there's never a large and dangerous forest fire? Why didn't all animal species develop civilization at the same time as humans, so that the human-caused extinction and extermination of most other species didn't happen?

It's because the jump from the less-favored to the more-favored state in a technological transition is complex and requires nontrivial adaptations which other groups would perhaps never develop, or would develop much more slowly. Dolphins can't make a civilization because they don't have access to h...
xAI announces Grok, beats GPT-3.5
by Nikola Jurkovic — 2023-11-05T22:11:15.274Z
Link: https://x.ai/

Some highlights:

Grok is still a very early beta product – the best we could do with 2 months of training – so expect it to improve rapidly with each passing week with your help.

I find it very interesting that they managed to beat GPT-3.5 with only 2 months of training! This makes me think xAI might become a major player in AGI development.

By creating and improving Grok, we aim to:

Gather feedback and ensure we are building AI tools that maximally benefit all of humanity. We believe that it is important to design AI tools that are useful to people of all backgrounds and political views. We also want empower our users with our AI tools, subject to the law. Our goal with Grok is to explore and demonstrate this approach in public.

On these benchmarks, Grok-1 displayed strong results, surpassing all other models in its compute class, including ChatGPT-3.5 and Inflection-1. It is only surpassed by models that were trained with a significantly larger amount of training data and compute resources like GPT-4. This showcases the rapid progress we are making at xAI in training LLMs with exceptional efficiency.

We believe that AI holds immense potential for contributing significant scientific and economic value to society, so we will work towards developing reliable safeguards against catastrophic forms of malicious use. We believe in doing our utmost to ensure that AI remains a force for good.
Disentangling four motivations for acting in accordance with UDT
by Julian Stastny — 2023-11-05T21:26:22.514Z

Introduction

In this post, I examine a number of different arguments for acting like an updateless agent. Concretely, I try to provide answers to the following question: What could move someone who isn’t already born a perfectly coherent UDT agent to adopt a UDT-like policy?

I take this perspective because I was not born as an idealized UDT agent, and I want the extent to which I’m willing to follow something like UDT (or endorse instructing an AI to do it) to be justified on non-tautological grounds. After all, doing ‘the updateless as opposed to the updateful thing’ by definition means doing something that, to first approximation, is bad by one’s current lights.

To gain clarity, I consider four ways of climbing the mountain of updatelessness. At the base, I assume that we start with non-indexical preferences and some pre-existing decision-theoretical intuitions, for instance ‘I want to cause good outcomes’ for those who are inclined towards CDT. The first three paths begin the journey from there: if we treat our current decision-theoretical intuitions as bedrock, will this lead us to act in UDT-like ways? The fourth approach I consider is to drop our previous decision-theoretical intuitions, attempting to jump up to UDT directly.

A caveat: none of the motivations I present here will receive the in-depth discussion they deserve. The role of this post is rather that of an opinionated guidebook: to make salient the fact that there exist different paths, the difficulties one is faced with by taking them, and that they end up at relevantly different UDT-like endpoints.

Four justifications for UDT-like policies

To set the stage, here are some deliberately vague definitions:

Updatelessness is the maxim of choosing the action you would have wanted to commit to ex-ante, i.e. before you learned everything you know now. What exactly this means depends on various factors, such as what you consider your ex-ante perspective to be, or what sorts of things you would/should have wanted to allow yourself to learn and act upon.

UDT

is a particular decision theory that was designed to fulfill the desideratum of updatelessness. Different versions of UDT have been proposed.[1]
A policy is a mapping from observations to actions. I call a policy UDT-like (in some decision problem) if it chooses the same actions as some updateless agent’s policy, though this doe

...
AI as Super-Demagogue
by RationalDino — 2023-11-05T21:21:13.914Z

Sam Altman recently said:

i expect ai to be capable of superhuman persuasion well before it is superhuman at general intelligence, which may lead to some very strange outcomes

I believe that he is absolutely right. Superhuman persuasion can be achieved by having LLMs consistently apply existing capabilities to known persuasion techniques. Some of these techniques are already proven to work at scale. Others have demonstrated effectiveness, and can more easily be applied at scale by AI. This makes superhuman persuasion a straightforward technology proposition.

I will look at this from the point of view of how AI can enable someone possessing the positions, skills, and desires necessary to attempt to create an authoritarian regime. In other words, I am describing something that is a near-term possibility without major advances in AI. Major advances seem likely, and would only make this picture worse.

I will also focus on techniques whose effectiveness has been proven by repeated human success. I will focus most of all on dictators and demagogues. Because he is so familiar, I will cite Donald Trump as an expert practitioner. This is not support for, or criticism of, him. I've aimed to have this read equally well whether or not you like him.

The information that I'm presenting is not new. Though it has come together for me as part of thinking about my blog. And I decided to post this as a response to, We are already in a persuasion-transformed world and must take precautions.

Talk to System 1

Dictators want a mass audience who is emotionally aligned with them. These people should want the dictator to be right. And ideally should find it painful to question the dictator. The result is followers who can't be convinced by rational argument.

This requires conditioning System 1. So you want to speak in a way that System 1 responds to, but doesn't activate System 2. Use your speeches to deliver a key emotional message over and over again.

The necessary verbal pattern involves lots of simple and repetitive language. Avoid questions. Questions activate System 2, and you don't want that. If you can, rearrange sentences so that the most impactful word comes last. Do not worry about logical consistency. If System 2 doesn't activate, logical gaps won't be noticed.

You can personally practice this speech pattern by studying dirty talking. If you're in a relationship, you can even pract...
EA orgs' legal structure inhibits risk taking and information sharing on the margin
by Elizabeth — 2023-11-05T19:13:56.135Z

What is fiscal sponsorship?

It’s fairly common for EA orgs to provide fiscal sponsorship to other EA orgs. Wait, no, that sentence is not quite right. The more accurate sentence is that there are very few EA organizations, in the legal sense; most of what you think of as orgs are projects that are legally hosted by a single org, and which governments therefore consider to be one legal entity.

The king umbrella is Effective Ventures Foundation, which hosts CEA, 80k, Longview, EA Funds, Giving What We Can, Asterisk magazine, Centre for Governance of AI, Forethought Foundation, Non-Trivial, and BlueDot Impact. Posts on the castle

also describe it

as an EVF project, although it’s not listed on the website. Rethink Priorities has a

program

specifically to provide sponsorship to groups that need it. LessWrong/Lightcone is hosted by CFAR, and have sponsored at least one project themselves (source: me. It was my project).

Fiscal sponsorship has a number of advantages. It gets you the privileges of being a registered non-profit (501c3 in the US) without the time-consuming and expensive paperwork. That’s a big deal if the project is small, time-limited (like mine was) or is an experiment you might abandon if you don’t see results in four months. Even for large projects/~orgs, sharing a formal legal structure makes it easier to share resources like HR departments and accountants. In the short term, forming a legally independent organization seems like a lot of money and effort for the privilege of doing more paperwork.

The downsides of fiscal sponsorship

…are numerous, and grow as the projects involved do.

The public is rightly suspicious about projects that share a legal entity claiming to be independent, so bad PR for one risks splash damage for all. The government is very confident in its belief that you are the same legal entity, so legal risks are shared almost equally (iamnotalawyer). So sharing a legal structure automatically shares risk. That may be fixable, but the fix comes at its own cost.

The easiest thing to do is just take fewer risks. Don’t buy retreat centers that could be described as lavish. And absolutely, 100%, don’t voluntarily share any information about your interactions with FTX, especially if the benefits to doing so are intangible. So some amount of value is lost because the risk was worth it for an individual ...
Eric Schmidt on recursive self-improvement
by Nikola Jurkovic — 2023-11-05T19:05:15.416Z
Link: https://www.youtube.com/watch?v=IeVY_Ag8JI8

Recently, Eric Schmidt gave a talk at Harvard called “Our AI Future: Hopes and Hurdles Ahead”. The entire talk is available

here

but there was one part that was interesting to me (around the 1:11:00 mark), and that is, his views on AI safety and trust in AI labs to stop scaling if recursive self-improvement starts happening. Emphasis my own.

The technical term is recursive self-improvement. As long as the system isn’t busy learning on its own, you’re okay. I’ll parrot Sam’s speech at OpenAI. [...]

In the next few years we’ll get to AGI [...] Some number of years after that, the computers are going to start talking to each other, probably in a language that we can’t understand, and collectively their superintelligence [...] is going to rise very rapidly. My retort to that is, do you know what we’re going to do in that scenario? We’re going to unplug them all.

The way you would do this if you were an evil person [...] is you would simply say to the computer: “Learn everything. Try really hard. Start right now.” So the computer starts learning. It learns about French, it learns about science, it learns about people, it learns about politics. It does all of that, and at some point, it learns about electricity and it learns that it needs electricity, and it decides it needs more. So it hacks into the hospital and takes the electricity from the hospital to give it to itself.

That’s a simple example of why this is so dangerous. Most people in the industry who thought about this believe that when you begin recursive self-improvement, there will be a very serious regulatory response because of these issues. And that makes sense to me.
Pivotal Acts might Not be what You Think they are
by Johannes C. Mayer — 2023-11-05T17:23:50.464Z

This article is mainly for people who have not read the pivotal act article on arbital or need a refresher. If you have, the most interesting section would probably be "Omnicient ML Researchers: A Pivotal Act without a Monolithic Control Structure".

Many people seem to match the concept of a "pivotal act" to some dystopian version of "deploy AGI to take over the world". 'Pivotal act' means something much more specific, though. Something, arguably, quite different. I strongly recommend you read the original article, as I think it is a very important concept to have.

I use the term quite often, so it is frustrating when people start to say very strange things, such as "We can't just let a powerful AI system loose on the world. That's dangerous!" as if that were the defining feature of a pivotal act.

As the original article is quite long let me briefly summarize what I see as the most important points.

Explaining Pivotal Act

An act that puts us outside of the existential risk danger zone (especially from AI), and into a position from which humanity can flourish is a pivotal act.

Most importantly that means a pivotal act needs to prevent a misaligned AGI from being built. Taking over the world is really not required per se. If you can prevent the creation of a misaligned AGI by creating a powerful global institution that can effectively regulate AI, then that counts as a pivotal act. If I could prevent a misaligned AGI from ever being deployed, by eating 10 bananas in 60 seconds, then that would count as a pivotal act too!

Preventing Misaligned AGI Requires Control

Why then, is 'pivotal act' often associated with the notion of taking over the world? Preventing a misaligned AGI from being built, is a tough problem. Efficively we need to constrain the state of the world such that no misaligned AGI can arise. To successfully do this you need a lot of control over the world. There is no way around that.

Taking over the world really means putting oneself into a position of high control, and in that sense, it is necessary to take over the world, at least to a certain extent, to prevent a misaligned AGI from ever being built.

Common Confusions

Probably, one point of confusion is that "taking over the world" has a lot of negative connotations associated with it. Power is easy to abuse. Putting an entity[1] into a position of great power can certainly go sideways. But I fail ...
The Assumed Intent Bias
by silentbob — 2023-11-05T16:28:03.282Z

Summary: when thinking about the behavior of others, people seem to have a tendency to assume clear purpose and intent behind it. In this post I argue that this assumption of intent quite often is incorrect, and that a lot of behavior exists in a gray area where it’s easily influenced by subconscious factors.

This consideration is not new at all and relates to many widely known effects such as the

typical mind fallacy

, the

false consensus effect

, black and white thinking and the concept of

trivial inconveniences

. It still seems valuable to me to clarify this particular bias with some graphs, and have it available as a post one can link to.

Note that “assumed intent bias” is not a commonly used name, as I believe there is no commonly used name for the bias I’m referring to.

The Assumed Intent Bias

Consider three scenarios:

When I quit my previous job, I was allowed to buy my work laptop from the company for a low price and did so. Hypothetically the company’s admins should have made sure to wipe my laptop beforehand, but they left that to me, apparently reasoning that had I had any intent whatsoever to do anything shady with the company’s data, I could have easily made a copy prior to that anyway. So they further assumed that anyone without a clear intention of stealing the company’s data would surely do the right thing then, and wipe the device themselves.
At a different job, we continuously A/B-tested changes to our software. One development team decided to change a popular feature, so that using it required a double click instead of a single mouse click. They reasoned that this shouldn’t affect feature usage of our users, because anyone who wants to use the feature can still easily do it, and nobody in their right mind would say “I will use this feature if I have to click once, but two clicks are too much for me!”. (The A/B test data later showed that the usage of that feature had reduced quite significantly due to that change)
In debates about gun control, gun enthusiasts sometimes make an argument roughly like this: gun control doesn’t increase safety, because potential murderers who want to shoot somebody will find a way to get their hands on a gun anyway, whether they are easily and legally available or not.[1]

These three scenarios all are of a similar shape: Some person or group (the admins; the development team; gun enthusiasts) make a judgme...