trentmkelly/LessWrong-43k huggingface dataset:
Should you announce your bets publicly?
by Ege Erdil — 2023-07-04T00:11:11.386Z

Over time, I've noticed many examples of people making bets that don't make much sense from a subjective profit-maximizing point of view. Without naming the people involved, here are some examples:

A lends B some amount of USD to be paid back in seven years, with an effective interest rate of 15% per year on the loan. This is construed as a bet on AI timelines: B thinks there's a high chance the world will end in seven years and won't have to pay back the debt, or that AI will drive substantial economic growth which will make the repayment of the loan much easier.

A and B make the following bet: B reads a book authored by A. If B changes his opinion about some important issue as a consequence of reading this book, B pays A an unspecified sum X. Otherwise, A pays B a sum of 2X. The context was such that I think A had no reason to think B would be convinced by their book, even aside from the issue of perverse incentives created by the bet.

A bets B at 1:4 odds (in A's favor) that AI will drive substantial economic growth over the next two decades. The point is that A thinks such an event is more than 1/(1+4) = 20% likely, while B thinks it's less than 20% likely, so the bet is meant to have a positive expected value from the standpoint of both counterparties.

A agrees to buy a Bitcoin from B for $1 million in 90 days, as a way of betting that the price of BTC will exceed $1 million in 90 days.

All of these examples have strange features that don't make much sense if we think of them as purely expected profit-maximizing actions. To enumerate;

B can just as easily get a loan denominated in USD for much less than 15% per year in interest. There are liquidity constraints, of course, but they are likely significantly less binding in the ordinary lending market than they are in the "making bets with acquaintances" market. Likewise, A can simply arbitrage the bet against said lending market as commercial lending rates are currently far below 15% per year, so their accepting this bet doesn't give us any information about their beliefs.

The bet is structured in a rather strange way that creates perverse incentives on B's part: he has a reason not to be convinced by A's arguments even if he in fact finds them convincing because now it costs him money to admit to doing so. Moreover, A has no guarantee that B will not lie about being convinced, and the conte

...
Ten Levels of AI Alignment Difficulty
by Sammy Martin — 2023-07-03T20:20:21.403Z

Image from https://threadreaderapp.com/thread/1666482929772666880.html

Chris Olah recently

released a tweet thread

describing how the Anthropic team thinks about AI alignment difficulty. On this view, there is a spectrum of possible scenarios ranging from ‘alignment is very easy’ to ‘alignment is impossible’, and we can frame AI alignment research as a process of increasing the probability of beneficial outcomes by progressively addressing these scenarios. I think this framing is really useful, and here I have expanded on it by providing a more detailed scale of AI alignment difficulty and explaining some considerations that arise from it.

The discourse around AI safety is dominated by detailed conceptions of potential AI systems and their failure modes, along with ways to ensure their safety. This

article by the DeepMind safety team

provides an overview of some of these failure modes. I believe that we can understand these various threat models through the lens of "alignment difficulty" - with varying sources of AI misalignment sorted from easy to address to very hard to address, and attempt to match up technical AI safety interventions with specific alignment failure mode scenarios. Making this uncertainty clearer makes some debates between alignment researchers easier to understand.

An easier scenario could involve AI models generalising and learning goals in ways that fit with common sense. For example, it could be the case that LLMs of any level of complexity are best understood as generative

frameworks over potential writers

, with Reinforcement Learning from Human Feedback (RLHF) or Constitutional AI (CAI) selecting only among potential writers. This is sometimes called

‘alignment by default’

.

A hard scenario could look like that outlined in

‘Deep Deceptiveness’

, where systems rapidly and unpredictably generalise in

ways that quickly obsolete previous alignment techniques

, and they also learn deceptive reward-hacking strategies that look superficially identical to good behaviour according to external evaluations, red-teaming, adversarial testing or interpretability examinations.

When addressing the spectrum of alignment difficulty, we should examine each segment separately.

If we assume that transformative AI will be produced, then the misuse risk associated with aligned transformative AI does not depend on how difficult alignm...
A Subtle Selection Effect in Overconfidence Studies
by Kevin Dorst — 2023-07-03T14:43:11.287Z
Link: https://kevindorst.substack.com/p/a-subtle-selection-effect-in-overconfidence

TLDR: The standard methods for studying overconfidence might be overestimating it. In asking people for confidence intervals—the narrowest intervals that they’re confident contain the true value—we’re selecting for the intervals that are most likely to be overestimates.

Fitz the fisherman assures you that all the fish in this cove are big—he’s been fishing in these waters for decades, and he’s never caught one smaller than 6 inches. You’re skeptical. What’s the first question you should ask?

“How big are the holes in your net?”

This is a classic example of a selection effect: if Fitz’s net has holes bigger than 6 inches, then he’d never catch small fish even if the cove is full of them.

Selection effects are a thorn in the side of calibration research: the subfield of psychology that tries to test when and why people are overconfident in their beliefs.

There are various approaches, but the most popular current method—known as “interval estimation”—asks people for intervals that they are confident contain the true value of some unknown quantity.

For example, “What proportion of Republicans think that racism still exists in America?” Take a second and make an interval that you’re 90%-confident contains the true proportion. Then check the answer in the footnote.[1]

Most people are overprecise: their intervals contain the true value far less than 90% of the time. Overprecision has been called the “mother of all biases” due to both its robustness and importance.

A previous post argued that many of the surprising findings surrounding overprecision make more sense once we realize that subjects aim to give answers that are informative relative to the question under discussion.

Today, I’d like to propose a complementary hypothesis: overprecision studies suffer from a subtle selection effect that (to my knowledge) has not been corrected for.

The History

First some history. Calibration research began in earnest in the 70s, when psychologists started to put the presuppositions of economic models to the test. Were people as rational and well-calibrated in their opinions as standard models of the free market assumed?

The original way to test for overconfidence was to ask people trivia questions, have them rate their confidence in their answers, and then record what proportion of answers were true at each level of confidence. If people are p...
Monthly Roundup #8: July 2023
by Zvi — 2023-07-03T13:20:05.614Z

As always, plenty of non-AI things are happening, most of which do not justify their own posts. This is where all of those go. Skimming for things relevant to your interests is encouraged.

Before I begin, a Point of Order: I plan to have a post later regarding the rate limitation and disruption on Twitter, which disrupts many features of Tweetdeck and means that if you aren’t logged in Twitter links no longer work. Adjustments will need to be made if the situation is not fixed very soon, despite my having no fear of ever hitting the 10k rate limit itself. It is not practical to go back and fix links or other decisions on already written material but I am adjusting the process going forward.

Bad News

RIP 538. Nate Silver leaving was always going to be a hard blow to recover from, as the models are the heart and soul of the entire model. It also depends on a deep commitment, visible to all, to fair play and avoiding putting a finger on the scale or using the scale as leverage. Alas:

Nate Silver: Very bad to do a Spanish Inquisition with pollsters based on their political orientation. I love my ex-colleagues (this is coming from a new guy they hired) but if this is their practice, hope ABC will stop use of 538 brand so it isn’t associated with me.

Washington Examiner: “Rasmussen must explain the nature of its relationship with several right- leaning blogs and online media outlets, which have given us reason to doubt the ethical operation of the polling firm,” demanded Elliott Morris, the editorial director of data analytics at liberal ABC News and FiveThirtyEight.

No. I have listened to many a Model Talk, with in-depth discussion of how to (among many other considerations) construct a polling aggregation systems that best resists partisan manipulation, despite the need to ‘play defense’ where the pollsters respond to your policies. Using logic like this is incompatible with such an effort.

Cocaine Bear, meet Government Warrantless Survey of Private Property Bear.

It is an important fact about banks that both they will pitch beyond atrocious products at you constantly, and also that no overloaded number of indicators, however explicit, that you know better, will get them to stop. Until McKenzie, I do begrudge the bank and the salesperson here, actively pitching a 2% CD right now is stealing. Worse, it’s wasting your time. The amount of idiocy I had...
Complex Signs Bad
by Evenstar — 2023-07-03T13:09:18.695Z

Complex signs bad.

Silly bullshit complex.

Good sturdy think not complex.

Doing not complex think help (past tense) Evenstar anxiety and depression.

Not complex signs help not complex think.

Make and doing (past tense) not complex signs.

But complex think with not complex signs good too.

Complex signs not sturdy for anxiety and depression.

Clever creative signs help Evenstar feel good.

Evenstar signs "Evenstar good!" and "Evenstar clever!" and "Evenstar creative!"

But for bad signs, "silly."

Evenstar signs help Evenstar sturdy.

Evenstar signs fun for Evenstar too!

Maybe help you sturdy too?

And because I know you probably won't take the above seriously:

Restricted Vocabulary as Thinking Aid

I recently invested in a discord emoji set for nonverbal communication, stealing the idea of a commboard from a plural community server I was in. Originally this was so that a particular persona of mine who was nonverbal could speak at all; however, I discovered several benefits on top of that.

I was able to structure a restricted vocabulary to discourage disordered (anxious, depressive) speech.

Having a safe vocabulary made it easier to plan and execute on plans without tripping over anxious or depressive thoughts.

Having the ability to express simple concepts to others under stress without having to type allowed me to manage extreme stress better (by making the above plans)

There's a mindset this promotes that helps you fix the "obvious thing" without getting tied up in knots over longer term planning.

It was fun to make my own custom emoji set and give myself words I thought were important, such as "fuck" and "bullshit"

Learning a "language" together with my girlfriends brought us closer.

Simple english and conlangs both capture aspects of this experience, but my set of words was heavily restricted (less than 100 to fit in a discord server's emoji slots) and this made me have to make a lot of hard choices about what was really important to me to express. Careful selection and tailoring of the emoji helped me decide what was important to say and what was not.

Reducing Disordered Speech

Saying bad things about yourself hurts - but anxiety and depression promote thoughts such as "I'm useless", "I'm worthless", "I'm trash", "I'm an awful person", "I'm broken".

When I first built my emoji set, I included a sign for "broken" in it, but quickly came to realize that it was actively harmful...
6/23
by Celer — 2023-07-03T06:30:07.676Z

Grab Bag

I read a thing? I can’t describe it more than that. There are very few words. Much happens. Strongly recommended.

I turned 29 this month. Apparently my chances of surviving were about that of not getting a 1 on a d20?

Politics and Policy

I reread An Essay on Crimes and Punishments, by Cesare Beccaria. Today, it is as powerful, though thankfully not quite as needed, as ever. If you’ve read Punishment Without Crime by Alexandra Natapoff, an excellent book looking at how misdemeanors are used to punish people who have done nothing worse than contempt of cop, this passage in particular may resonate.

No man can be judged a criminal until he be found guilty; nor can society take from
him the public protection, until it have been proved that he has violated the conditions on which it was granted. What right, then, but that of power, can authorise the punishment of a citizen, so long as there remains any doubt of his guilt? The dilemma is frequent. Either he is guilty, or not guilty. If guilty, he should only suffer the punishment ordained by the laws, and torture becomes useless, as his confession is unnecessary. If he be not guilty, you torture the innocent; for, in the eye of the law, every man is innocent, whose crime has not been proved.

Can you understand the modern jail system, for anyone except the rich and powerful, as anything other than a form of torture that we justify? It may be that it is expedient, and it may be that some of the harm of torture is the infliction of suffering for that sake alone, but Beccaria reminds us that it is not the inherent wrongness of harming another being in some ways that is contrary to a good society. It is the entire institutionalization of extra-legal punishment of the accused, simply for being accused.

War on the Rocks has a summer reading list.

There was a brief period in which I didn’t realize that RFK Jr., the anti-vax presidential candidate who boasts about an assisted bench-press of 115lb, didn’t live in LA. Of course he lives in LA. How could he be anywhere else?

Shared politics by left-wing activists has, in my view, constructed a common narrative around oppression and liberation that many sincerely believe. It's a solid moral framework. There’s just a minor issue that not all of the people understood as “the oppressed” buy into it, and so you get conservative religious communities pushing against queer liberatio...
My Central Alignment Priority (2 July 2023)
by Nicholas / Heather Kross — 2023-07-03T01:46:01.764Z

Epistemic status: The further you scroll, the more-important the points are.

I continue my deconfusion from my alignment timeline post. Goal: figure out which sub-problem to work on first in AI alignment.

Quick Notation

1 = steering

2 = goal

1.1k no steering found

1.2k poor steering found

1.3g steering lets us do pivotal

1.4g steering + existing goal inputs

1.5g steering not needed, alignment by default

2.1k no goal found, even with steering

2.2k poor goal found, combined with steering

2.2g goal is good/enough

The Logic

Let's say that my effort single-handedly changed the relative balance of research between 1 and 2. So we ignore scenarios where my work doesn't do anything. (By same intuition, we ignore 1.5g, since that doesn't require effort.)

If I research 1 (first) and X happens, what happens?

I find bad ideas I think are good. --> I overconfidently promote it, wasting time at best and causing 1.2k at worse.

I find good ideas. --> 1 is solved --> 1.3, 1.4, or 2 must happen.

1.3g: I, or a group I trust, must be able to do pivotal act before 1's solution leaks/is independently re-discovered.

1.4g: Good ending is easy to achieve by quickly coding an implementation of 1's solution.

2: I, or a group I trust, must be able to solve 2 before 1's solution leaks/is independently re-discovered.

If I research 2 (first) and X happens, what happens?

I find bad ideas I think are good. --> I overconfidently promote it, wasting time at best and causing 2.2k at worst.

I find good ideas. --> 2 is solved --> 1 must also be solved, for 2.2g to happen.

1: I, or a group I trust, must be able to solve 1.

The Bottom Line

I, personally, right now, should have the key research focus of "solve problem 2".

If I get interesting ideas about problem 1, I should write them down privately and... well, I'm not sure, but probably not publish them quickly and openly.

This will be the case until/unless something happens that would have made me change my above logic.

Some things that have not happened, but which I would expect to change my mind on the above points:

Problem 1 or problem 2 gets obviously-solved. --> Jump to the other respective branch of the logic tree.

A global AI pause actually occurs, in a way that actually constrains Anthropic and OpenAI and DeepMind and Meta s.t. AGI timelines can be pushed further out. --> Tentatively prioritize working

...
My Alignment Timeline
by Nicholas / Heather Kross — 2023-07-03T01:04:07.935Z

Epistemic status: If I don't post this now, I'll have a harder time treating it as something to build on.

Which chapter do we end on?

Which chapter of doomsday are you on, right now?

I think of AI-alignment-going-well as a process, where AI-causes-human-extinction-or-worse is an event that will by-default interrupt (and end) that process.

Many cruxes about AI alignment subproblems, reduce to "Which part do you think is The Part That Kills Us?".

Consider the following game tree:

Find a way to steer AI at all, as it reaches high capability levels.

Part That Kills Us: We don't find this.

Part That Kills Us: We find something that looks like this, and implement it, but it actually doesn't survive the sharp-left-turn.

Good Ending: Steerability solves most of the problem, and we can then do a pivotal act.

Good Ending: The capabilities "sharp left turn" doesn't break our steering, and we can just apply existing human-goal-input-methodologies to steerable AI.

Good Ending: We don't need better "steering"; alignment by default, modulo some non-theoretic engineering advancements.

Once AI is steerable, give it goals (or goal-replacing structures) that, at least, don't involve extinction- or suffering-risks, and hopefully lead to really good outcomes.

Part That Kills Us: We don't do this, even given a steerable AI.

Part That Kills Us: We find something that looks like this, and implement it, but it actually causes human extinction/suffering-risks instead.

Good Ending: The goal is good, or at least good enough.

So a "Part That Kills Us" is a leaf on the game-tree that ends badly, while a "Good Ending" is a leaf that avoids ending-badly.

The good news: a non-zero number of people seem to be working on each individual step in this game tree.

The bad news: they've attached their substep to the rest of their AI beliefs, and this broke the field.

Worlds Where Some Chapters Don't Matter

I want to clarify some relationships among branches of this game tree. This will help deconfuse me, and maybe other people also.

On the Good Ending 1.5 (alignment by default, appears as "1.e" on LessWrong), further AI alignment work is not needed. However, humanity is already pretty good at such "mere engineering" issues. Worlds where we should worry most about AI alignment, are the worlds where mere engineering doesn't work. This, in turn, implies that AI alignment should

...
Douglas Hofstadter changes his mind on Deep Learning & AI risk (June 2023)?
by gwern — 2023-07-03T00:48:47.131Z
Link: https://www.youtube.com/watch?v=lfXxzAVtdpU&t=1763s

A podcast interview (posted 2023-06-29) with noted AI researcher Douglas Hofstadter discusses his career and current views on AI (via Edward Kmett), and amplified to David Brooks.

Hofstadter has previously energetically criticized GPT-2/3 models (and deep learning and compute-heavy GOFAI).
These criticisms were widely circulated & cited, and apparently many people found Hofstadter a convincing & trustworthy authority when he was negative on deep learning capabilities & prospects, and so I found his most-recent comments (which amplify things he has been saying in private since at least 2014) of considerable interest.

This interview (EDIT: and earlier material, it turns out), appears to have gone under the radar, perhaps because it's a video, so below I excerpt from the second half where he discusses DL progress & AI risk:

Q: ...Which ideas from
GEB are most relevant today?

Douglas
Hofstadter: ...In my book, I Am a
Strange
Loop, I tried to set forth what
it is that really makes a self or a soul. I like to use the word
"soul", not in the religious sense, but as a synonym for "I", a
human "I", capital letter "I." So, what is it that makes a human
being able to validly say "I"? What justifies the use of that
word? When can a computer say "I" and we feel that there is a
genuine "I" behind the scenes?

I don't mean like when you call up the drugstore and the
chatbot, or whatever you want to call it, on the phone says,
"Tell me what you want. I know you want to talk to a human
being, but first, in a few words, tell me what you want. I can
understand full sentences." And then you say something and it
says, "Do you want to refill a prescription?" And then when I
say yes, it says, "Gotcha", meaning "I got you." So it acts as
if there is an "I" there, but I don't have any sense whatsoever
that there is an "I" there. It doesn't feel like an "I" to me,
it feels like a very mechanical process.

But in the case of more advanced things like
ChatGPT-3 or
GPT-4, it feels like there
is something more there that merits the word "I." The question
is, when will we feel that those things actually deserve to be
thought of as being full-fledged, or at least partly fledged,
"I"s?

I personally worry that this is happening right now. But it's
not only happening right now. It's not just that certain things
that are coming about are similar to human consciousness or
human

...