trentmkelly/LessWrong-43k huggingface dataset:
I believe some AI doomers are overconfident
by  — 2022-12-20T17:09:23.325Z

Just up front: I have no qualifications on this so adjust accordingly. I teach AP Calc/AP Stats if you want to know my deal. Putting this down because half the time I'm reading a post I'm thinking to myself "I wish I knew what this person's deal is" :)

People who believe there is a >50% possibility of doom in the next 50 years or so strike me as overconfident. Just to say the general public is obviously way underestimating the risk (0% too low lol), but I believe many people on this site are overestimating the risk.

The complexity of the system is just so high. How can we predict what a superintelligence that hasn't even been created yet will behave? I understand that it's reasonable to assume it will want to accumulate resources, eliminate threats (us lol) etc., but how can anyone be for instance 90%+ sure that it will end with us dead?

There are so many links of reasoning that all have at least a small chance of going awry. Just spit-balling off the top of my head-- I'm sure all of these can be refuted, but like are you 100% sure? Just to say these specifics are not the point of my argument: I'm just arguing that there are a million things that could in theory go wrong, and even if each is unlikely, it's kind of a swiss-cheese defense against doom:

Maybe when a certain level of intelligence is reached, consciousness comes online, and that affects the behavior of the system in an unpredictable way.
Maybe alignment works!
Maybe the system is unbelievably super-intelligent, but for some reason true social reasoning is basically impossible for LLMs and the like, and we need to go down a different, distant path before that becomes possible. We can still easily trick it!
Maybe superintelligence is subject to rot. Many complex systems just kind of decay, and maybe for some reason this intelligence is unable to constantly update its code and maintain itself perfectly.
Maybe it's actually worse than it seems, but fledgling AIs go crazy in 2035 and kill 50 people in a sufficiently brutal way that the world wakes up and we destroy every computer we can see with a sledgehammer before scaled up evil-AIs are possible.

It just seems like there a million things that could potentially go wrong. If you ask me to predict what would happen if a hungry tiger came into a room with me and three of my friends, I couldn't do it...
Note on algorithms with multiple trained components
by Steven Byrnes — 2022-12-20T17:08:24.057Z

Example 1:

consider a GAN. There’s a generator and a discriminator. As an intuitive mnemonic, we can say

The “purpose” of the generator is to trick the discriminator,
The “purpose” of the discriminator is to not get tricked by the generator.

(Relatedly, people will say “the generator is trained to trick the discriminator”, etc.)

…But (I hope) everyone knows that these bullet points are only a mnemonic.

The one and only real “purpose” of the whole system and everything in it is to generate cool images that we like, and get our papers into NeurIPS or whatever.

And indeed, I think everyone who uses GANs is aware that it’s possible for a programmer to make the discriminator “better” (when narrowly viewed as having a “purpose” of not getting tricked by the discriminator), but with the direct result of making the whole system worse at generating cool images. For example, if there were a code-change that made the discriminator perfect at discriminating, then there would be no gradient for training the generator, and the whole system would be useless.

So we shouldn’t take those bullet-point mnemonics too literally.

Example 2:

In actor-critic RL, people sometimes say:

The “purpose” of the value function is to approximate future rewards [or discounted sum of future reward, or whatever].

…But that’s also just a mnemonic. The one and only real “purpose” of the whole RL system (of which the value function is just one part) is that it does whatever we want the RL system to do, e.g. win at chess, get our papers into NeurIPS, build us a luxury gay space communist utopia, etc.

So it’s at least conceivable that some algorithmic change would make the value function into a better approximation of the discounted sum of future rewards, yet make the RL agent worse at doing things that we want it to do.

Actually, this particular example is not merely “conceivable”, but expected, thanks to wireheading. If the value function is used to assess which plans are good versus bad, and the value function is a perfect approximation of expected future reward, then you’re almost guaranteed to get an AI that is trying to wirehead.

(I myself am a model-based RL agent (I claim), and I don’t want to wirehead, and I claim that this is directly related to my internal value function issuing very inaccurate predictions of the future reward associated with wireheading. Details in footnote.[1])

So anyway, I expect ou...
Marvel Snap: Phase 2
by Zvi — 2022-12-20T14:50:00.460Z

[NOTE: This is a pure gaming post, if you’re not into that please do skip it.]

Previously: Marvel Snap Review: Phase 1

At the end of my Phase 1 review, I removed Marvel Snap from my phone.

I am very happy with that decision.

I did then continue to play the game on my computer, via Steam.

I am torn about that decision.

On the one hand, a lot of the games continue to be super interesting, involve interesting decisions, and allow fun to be had. I’ve learned a bunch about games and game design, both the good and the bad. I get to take fun quick breaks when I want one.

On the other hand, the ability to jump into such games is distracting, thinking about what one might do or how one might optimize the game is distracting, and the game is, shall we say, not the best use of my marginal time and thinking. Over time, the game changes less as the rewards get increasingly stingy, as they always do in such games. I am in no danger of spending any money, or getting too ruthless about maximization, but the brain space occupation even without my phone remains real.

Now that the days have gone by, and enough credits have been spent that Pool 1 has become Pool 2 and now Pool 3, it seems worth checking in once again to see what observations seem salient.

My assessment of the game has not much changed – it is who we thought it was.

Mostly, anyway.

The Correct Response to Snaps Is To Fold a Lot

If someone snaps, you should probably fold.

Have I won plenty of games where my opponent snapped and then it turned out to be either a misclick, a bluff, a bot or something really weird? Yes, of course.

It is still not the way to bet, especially if it’s not turn 1 or 2. When the vast majority of players snap, they have a damn good reason for doing so. At a minimum, their deck is going to Do its Thing, whatever it is. Often they have something that will blow you out of the water.

Game theory optimal play requires calling a substantial portion of the time on most boards. Ignore that. People almost never bluff. If it wouldn’t be in your opponents’ best interest to bluff you, you aren’t folding enough. If there is a card or sequence they could plausibly have that wins them the game, that’s it. Game over.

One good reason for this is that folding is no fun. Another is that you’ll never know. You never find out what cool thing your opponent had in mind. Presumably at the highest levels this gets...
(Extremely) Naive Gradient Hacking Doesn't Work
by ojorgensen — 2022-12-20T14:35:33.591Z

Tldr

A super simple form of gradient hacking might be to perform badly on any change to the value of the model's mesa-objective, in order to preserve the mesa-objective. I use some simple properties of SGD to explain why this method of gradient hacking cannot work.

Thanks for Jacob Trachtman, Jakub Kraus, and Francis Priestland for feedback. Thanks also to Evan Hubinger for responding to my questions.

Assumed Knowledge

I think this post will be most valuable for those who have some familiarity with the alignment problem (in particular, understanding of inner misalignment, mesa-optimisation). Having done something like AGISF would be more than sufficient for reading this.

As an introduction to mesa-optimsation I’d recommend

Risks from Learned Optimization: Introduction

by Evan Hubinger.

A naive description of gradient hacking

Gradient hacking is the hypothesised phenomenon whereby a model becomes aware of its training process, and then tries to influence the training process in some way that helps it to pursue its own objective (its mesa-objective).

If possible, one of the simplest ways a model could try to continue pursuing its mesa-objective is to ensure that the mesa-objective is fixed by an iteration of SGD. A hypothetical example of this first introduced by

Evan Hubinger

, which he no longer endorses, is as follows[1].

Suppose a model can check if its objective satisfies some criterion, and perform extremely badly if the criterion doesn’t hold. This means that the model can prevent gradient descent from changing its objective, since any change to its objective would lead to extremely bad performance.

I’ll call this strategy Naive Gradient Hacking, or NGH for short. I’m going to briefly explain why gradient hacking cannot look like this, by looking at how SGD works. For an exploration of what gradient hacking could look like, I’d recommend checking out

Richard Ngo's

posts

.

How does SGD Work?

Let x be some input to our model f(⋅|θ), which is a Neural Network, with parameter vector given by θ=(θi)1≤i≤N. Assume that SGD is being used to minimise some loss function L(f(x|θ)) of f.[2]

There are four important aspects of SGD that I want to convey here, which are necessary to understand the problems with Naive Gradient Hacking.

Point 1

First, recall that the change to any parameter θi due to a single iterat...
An Open Agency Architecture for Safe Transformative AI
by davidad — 2022-12-20T13:04:06.409Z

Edited to add (2024-03): This early draft is largely outdated by my ARIA programme thesis, Safeguarded AI. I, davidad, am no longer using "OAA" as a proper noun, although I still consider Safeguarded AI to be an open agency architecture.

Note: This is an early draft outlining an alignment paradigm that I think might be extremely important; however, the quality bar for this write-up is "this is probably worth the reader's time" rather than "this is as clear, compelling, and comprehensive as I can make it." If you're interested, and especially if there's anything you want to understand better, please get in touch with me, e.g. via DM here.

In the Neorealist Success Model, I asked:

What would be the best strategy for building an AI system that helps us ethically end the acute risk period without creating its own catastrophic risks that would be worse than the status quo?

This post is a first pass at communicating my current answer.

Bird's-eye view

At the top level, it centres on a separation between

learning a world-model from (scientific) data and eliciting desirabilities (from human stakeholders)
planning against a world-model and associated desirabilities
acting in real-time

We see such a separation in, for example, MuZero, which can probably still beat GPT-4 at Go—the most effective capabilities do not always emerge from a fully black-box, end-to-end, generic pre-trained policy.

Hypotheses

Scientific Sufficiency Hypothesis: It's feasible to train a purely descriptive/predictive infra-Bayesian[1] world-model that specifies enough critical dynamics accurately enough to end the acute risk period, such that this world-model is also fully understood by a collection of humans (in the sense of "understood" that existing human science is).

MuZero does not train its world-model for any form of interpretability, so this hypothesis is more speculative.
However, I find Scientific Sufficiency much more plausible than the tractability of eliciting latent knowledge from an end-to-end policy.

It's worth noting there is quite a bit of overlap in relevant research directions, e.g.

pinpointing gaps between the current human-intelligible ontology and the machine-learned ontology, and
investigating natural abstractions theoretically and empirically.

Deontic Sufficiency Hypothesis: There exists a human-understandable set of features of finite trajectories in such a world-model, taking values

...
Under-Appreciated Ways to Use Flashcards - Part I
by Florence Hinder — 2022-12-20T12:43:31.387Z
Link: https://thoughtsaver.ghost.io/under-appreciated-ways-to-use-flashcards/

By Tereza Ruzickova

Those who know me well sometimes call me a “flashcard freak,” - and they are not wrong.

I first discovered flashcards at the end of high school, as I was getting frustrated with my own forgetting curve. I learned so many interesting facts as a student, but without regular use of those memories, time erased them faster than

the vacuum cleaner from Inside Out.

All of this changed when I discovered spaced repetition software, and it became an integral part of my learning. However, up until recently, I haven't shared this with many people - I thought it would just sound nerdy and weird!
At EAG in London, I ended up talking to Flo and the team at

Thought Saver.

I was fascinated to discover that they were as excited about spaced repetition as me. In particular, we bonded over all the innovative ways in which flashcards can improve people's lives. Traditionally, people use them mostly to improve their memory of foreign languages or general knowledge trivia. But wouldn't it be even better to use them as a learning tool for what's most valuable in life? Flo described her vision of helping people learn about well-being and decision-making. I also had many thoughts on utilizing it for better mental health and relationships. This got us inspired to collaborate!
How I use flashcards

Flashcards have been with me through thick and thin - I used them to prepare for my Oxford application, for all my exams and job interviews, and even for my mental health! As a result, I've accumulated more than 40,000 cards over the last ten years.

Why would I do such a freakish thing? Well, where do I even start?

1.

Because it works.

Active retrieval of information is one of the most effective methods for memorizing anything. In the scientific literature, you can find it under terms such as “the testing effect” or “desirable difficulty during learning” - Basically, it refers to any effortful recall of information. While passive learning (such as listening to a podcast episode) can make us feel like we will remember the information in the future, we won't know this for sure until we put our knowledge to an active test. While flashcards are not the only way to practice this, they do make it particularly convenient and systematic.

2. Because it makes learning so much more efficient. With flashcards, you only spend time learning what you currently don't know very well. As you recall...
EA & LW Forums Weekly Summary (12th Dec - 18th Dec 22')
by Zoe Williams — 2022-12-20T09:49:51.463Z

Supported by Rethink Priorities
This is part of a weekly series summarizing the top posts on the EA and LW forums - you can see the full collection here. The first post includes some details on purpose and methodology. Feedback, thoughts, and corrections are welcomed.

If you'd like to receive these summaries via email, you can subscribe here.

Podcast version: prefer your summaries in podcast form? A big thanks to Coleman Snell for producing these! Subscribe on your favorite podcast app by searching for 'EA Forum Podcast (Summaries)'. More detail here.
Author's note: I'm heading on holidays, so this will be the last weekly summary until mid-January. Hope you all have a great end of year!

Top / Curated Readings

Designed for those without the time to read all the summaries. Everything here is also within the relevant sections later on so feel free to skip if you’re planning to read it all. These are picked by the summaries’ author and don’t reflect the forum ‘curated’ section.

Announcing WildAnimalSuffering.org, a new resource launched for the cause

by David van Beveren

Vegan Hacktivists

released

this website

, which educates the viewer on issues surrounding Wild Animal Suffering, and gives resources for getting involved or learning more. Their focus was combining existing resources into something visually engaging and accessible, as an intro point for those interested in learning about it. Please feel free to share with your networks!

The winners of the Change Our Mind Contest—and some reflections

by GiveWell

First place winners of GiveWell’s contest for critiques of their cost-effectiveness analyses:

GiveWell’s Uncertainty Problem

by Noah Haber: The author argues that without properly accounting for uncertainty, GiveWell is likely to allocate its portfolio of funding suboptimally, and proposes methods for addressing uncertainty.

An Examination of GiveWell’s Water Quality Intervention Cost-Effectiveness Analysis

by Matthew Romer and Paul Romer: The authors suggest several changes to GiveWell's analysis of water chlorination programs, which overall make Dispensers for Safe Water's program appear less cost-effective.

They assigned two first place winners due to the quality of submissions, in addition to 8 honorable mentions, and $500 prizes for all others of the 49 entries meeting contest criteria.

GiveWell think the contest was worth doing, providing both...
[link, 2019] AI paradigm: interactive learning from unlabeled instructions
by the gears to ascension — 2022-12-20T06:45:30.035Z
Link: https://jgrizou.github.io/website/projects/thesis

Sent by a friend, I thought this was an interesting reference folks on lesswrong should know about. I have not attempted to grok this paper. If anyone thinks I should, please reply and let's schedule a time to read and discuss it in order to explain it to each other and catch mistakes; I have a very hard time doing anything without someone else agreeing with me that it's important enough to socialize about, if I could I would join a full time study group network. My agency is quite weak.

Providing you neither know the game of chess nor the French language, could you learn the rules of chess from a person speaking French? In machine learning, this problem is usually avoided by freezing one of the unknowns (e.g. chess or french) during a calibration phase. During my PhD, I tackled the full problem and proposed an innovative solution based on a measure of consistency of the interaction. We applied our method to human-robot and brain-computer interaction and studied how humans solve this problem. This work was awarded a PhD prize handed by Cédric Villani (2010 Fields Medal).

Approach

Can an agent learn a task from human instruction signals without knowing the meaning of the communicative signals? Can we learn from unlabeled instructions? [1]

The problem resemble a chicken-and-egg scenario: to learn the task you need to know the meaning of the instructions (interactive learning), and to learn the meaning of the instructions you need to know the task (supervised learning).

However, a common assumption is made when tackling each of the above problems independently: the user providing the instructions or labels is acting consistently with respect to the task and to its own signal-to-meaning mapping. In short, the user is not acting randomly but is trying to guide the machine towards one goal and using the same signal to mean the same things. The user is consistent.

Illustration of consistency. The same dataset is labelled according to 3 different tasks. The left labelling is the most consistent with the structure of the user signals - indicating the user was probably teaching that task.

Hence, by measuring the consistency of the user signal-to-meaning mapping with respect to different tasks, we are able to recover both the task and the signal-to-meaning mapping, solving the chicken-and-egg problem without the need for an explicit calibration phase. And we do so while

...
[Fiction] Unspoken Stone
by Gordon Seidoh Worley — 2022-12-20T05:11:23.231Z

Author's note: I got the seed of an idea for a story. As an exercise, I wrote it down. I have no expectation of returning to this; I have a larger writing project I care about more. But, it was fun to write, so maybe you'll find it fun to read.

"Dr. Passo, thank you for coming."

The young man extended a slender hand from his wrinkled, grey suit. Alex took it, taking care not to crush it with his callused mitts.

"I'm Johnson. Been working with the Firm for a few months now. How was your flight from Bologna? Sorry we couldn't get you a seat in first class."

Alex muttered something unintelligible about how it was fine.

"Well, if you're ready to get started, follow me."

Johnson led Alex to an elevator at the far end of the empty warehouse. The kid used a key card to open the elevator. Once they were inside, Johnson placed his hand on a scanner. The elevator beeped and closed. Alex's stomach jumped as the car descended quickly.

"So, just what is it you want me to look at?" asked Alex.

"All in good time, Dr. Passo. We can discuss specifics once we're in the Cage."

After what felt like an eternity, the elevator beeped again and the door opened. Two armed guards stood before a smooth, metal door.

"ID," said the one on the left.

Johnson flashed his badge. The guard looked at Alex. Not sure what to do, Alex pulled out his passport and held it up. The guard nodded and seemed satisfied. The other guard reached back, pressed a button, and the door slid open.

Johnson led Alex inside to a narrow room where they met another guard standing behind a small table. There was something that looked like a metal detector and another door at the far end.

"Empty your pockets and leave everything out here. If you try to smuggle anything inside the scanner will pick it up. I'll give you a ticket so you can reclaim your stuff when you leave."

Alex thought this was all a bit much to look at an artifact, but he complied. He didn't know much about Halstead Holding Firm, but so far had gathered that they were wealthy and secretive. Given they were paying him more than he could make in five years at the University, he figured it best to get along and not ask too many questions.

"Pockets empty?" asked Johnson, already knowing the answer. "Right this way then."

Johnson's outstretched arm pointed Alex towards the scanner. Alex stepped through it. Nothing happened. Then the door opened.

"Go on through, Dr. Passo...