trentmkelly/LessWrong-43k huggingface dataset:
Yet more UFO Betting: Put Up or Shut Up
by MoreRatsWrongReUAP — 2023-08-08T17:50:44.556Z

(Edit: I am not the user

RatsWrongAboutUAP

.)

I'm offering a new bet along similar lines to Yudkowsky's bet with the user

RatsWrongAboutUAP

in

this thread

last month. My bet has these similarities (here added explicitly after some confusion in the comments):

I will pay out immediately upon our agreement to the bet, whereas the other party only needs to pay out if and when they lose the bet. I am not requiring the other party to put their funds in escrow.
I am not asking for the other party to reveal private financial information (though doing so may allow me to give better odds than 150:1)
I am willing to make multiple bets with different people.
The odds I'm offering are 150:1.

My bet has these differences:

The time frame will be 3 years from the day the bet is agreed to.
The bet will resolve in my favor if a non-prosaic origin of the existing evidence, or new evidence, is confirmed in this time frame. Edit: the requirement of confirmation of evidence (by the same standards that Yudkowsky agreed to) applies both to existing evidence and any evidence that may emerge during the time interval.
My wager is $1000. If I make multiple bets, this will be the total wager.

EDIT: If you can document liquid assets, I will go down to 100:1. The limit to odds I will accept is at 75:1, but that would only be if the other wager was put in escrow.

I'd like to discuss details like credibility of payback, but first I want to see if there's any interest. My suspicion is that LW users are more afraid to make this bet now than they were in July, but we'll see.
AISN #18: Challenges of Reinforcement Learning from Human Feedback, Microsoft’s Security Breach, and Conceptual Research on AI Safety
by Dan H — 2023-08-08T15:52:07.764Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-18

Welcome to the AI Safety Newsletter by the

Center for AI Safety

. We discuss developments in AI and AI safety. No technical background required

Subscribe here to receive future versions.

Challenges of Reinforcement Learning from Human Feedback

If you’ve used ChatGPT, you might’ve noticed the “thumbs up” and “thumbs down” buttons next to each of its answers. Pressing these buttons provides data that OpenAI uses to improve their models through a technique called reinforcement learning from human feedback (RLHF).

RLHF is popular for teaching models about human preferences, but it faces fundamental limitations. Different people have different preferences, but instead of modeling the diversity of human values, RLHF trains models to earn the approval of whoever happens to give feedback. Furthermore, as AI systems become more capable, they can learn to deceive human evaluators into giving undue approval.

Here we discuss a new paper on the problems with RLHF and open questions for future research.

How RLHF is used to train language models. Large language models such as ChatGPT, Claude, Bard, LLaMA, and Pi are typically trained in two stages. During the “pretraining” stage, the model processes large amounts of text from the internet. The model learns to predict which word will appear next in the text, which provides it a broad grasp of grammar, facts, reasoning abilities, and even biases and inaccuracies present in the training data.

After pretraining, models are “fine-tuned” for particular tasks. Sometimes they’re fine-tuned to mimic demonstrations of approved behavior. Reinforcement learning from human feedback (RLHF) is a technique where the model provides an output and a human provides feedback on it. The feedback is relatively simple, such as a thumbs up or a ranking against another model output. Using this feedback, the model is refined to produce outputs which it expects would be preferred by human evaluators.

For a more detailed technical explanation of RLHF, see here. Alternatively, for a lighter treatment of the topic, see the “shoggoth” meme below.

The “shoggoth” meme depicts GPT-3 as an alien monster unlike any human being. Fine-tuning with RLHF masks the shoggoth.

Aligned with whom? Humans often disagree with one another. Our different values, cultures, and personal experiences shape our beliefs and desires. Instead of modeling different perspectives and considerin...
Beginner's question about RLHF
by  — 2023-08-08T15:48:24.118Z

I don't fully understand RLHF, so let me just put my idea out there and someone can tell me how I am confused.

Assume that RLHF can in fact "align" an otherwise dangerous system. Here's my question: You train the system, and then you start applying RLHF, right? Doesn't the dangerous system already exist, then, before the RLHF is applied? If so, isn't that dangerous?

Anyway I don't know any technical details so I assume I simply don't understand how these things work.
My Trial Period as an Independent Alignment Researcher
by Bart Bussmann — 2023-08-08T14:16:35.122Z

In the past two months, I have tried out what is like to be an independent alignment researcher. My goals were to figure out if this path is something I would like to do, whether I'm a good fit, which research areas are most promising for me, and whether I feel like I can actually contribute something to the alignment problem.
My approach was to dive into different alignment subfields. In each subfield, I aimed to identify an open problem, work for about a week or two on this problem, and track feelings of hope and progress. This post is a reflection of this two-month trial period.

Being an independent researcher is great

Seriously. Being an independent researcher is fantastic. Every day, you have the opportunity to work on whatever is on your mind, whatever you find most interesting, and whatever you think has the most impact. This is truly living the dream. Millions of intellectuals around the world aspire to this life and aim to do this whenever they get tenure or retire. If you can get funding to do independent alignment research, you can live this life now.

Being an independent researcher is hard

That said, being an independent researcher isn't easy. Identifying what you want to work on is hard. Deciding when to switch research directions is hard. Deciding whether to start a new project or first finish an old project is hard. Motivating yourself and battling procrastination alone is hard. Uncertainty about whether you'll get a new grant after the current one is hard.

No one is telling you what to work on, whether you've done enough for the day, or if you're headed in the right direction. You have to weigh your personal interests with what you think has the most expected value. You might feel like you're wasting valuable EA money. Doing independent research is difficult to explain to friends and family. You have to decide for yourself whether it's okay to take a day off or go on a holiday. You might feel guilty when you do. You have to decide how to divide your time between learning, thinking, experimenting, and writing.

But you can make it easier for yourself

Cowork with other people. During the first month, I worked from home, and during this second month, I got a desk in the local EA office. This made a lot of difference. Coworkers will force you to take breaks, and breaks in which you can chat with someone else are much more rejuvenating. I...
4 types of AGI selection, and how to constrain them
by Remmelt — 2023-08-08T10:02:53.921Z

This is a quick write-up of part 2 of my VAISU talk. See also part 1, and longer post.
td;lr A rough framework for understanding the ways AGI gets selected for,
and the ways communities can constrain that selection by acting now.

Selection toward AGI

Types of feedback loops through which AGI[1] functionality is gradually selected:

Machine learning
Code internally optimised given data for continued capacity to optimise for next data inputs (toward instrumental convergence).

Human tinkering
Workers tinkering with code and hardware for whatever works, as new tools used for tinkering... (toward larger neural networks and robotics that somehow work).

Institutions competing
Corporate uses of AI that extract more profit/social influence/resources, which are reinvested into more machinery, allowing more uses… (toward multi-polar traps).

Artificial selection
Components' external effects selected for feeding back into their continued increased existence (toward substrate-needs convergence).

These loops are mutually reinforcing, in where and at what scale, they select for AGI.[1]

1. (within hardware) and 2. (within institutions) are about internal selection.

3. (via markets/social hierarchies/infrastructure) and 3. (via environmental effects) are about external selection.

Prerequisites for AGI

Prerequisites[2] for that selection to occur:

Data (laundering)
pirated from original authors, and surveyed from communities and their spaces.
eg. Text and images scraped from creatives. Exchanges scraped from citizens.
Data from data workers are underpaid if paid at all.

Workers (exploitation)
misled to dedicate their talent, and/or eventually replaced.
eg. Talented ML researchers are recruited with tech utopia claims. Software engineers are hired to set up infrastructure, and then fired. A much larger number of data workers and creatives are also misled to take on jobs for training of ML models.

Misuses (of AI services)
lobbied for by corporations to gain profit and power.
eg. General-Purpose AI is a marketing term implying that a given model can be and should be used for any purpose. While profitable for Big Tech, this would violate basic system-safety practices in eg. medical, automotive and industrial engineering.

Compute (pollution)
polluting, energy-sucking hardware infrastructure.
eg.

...
Notice your everything
by metachirality — 2023-08-08T02:38:39.974Z

[epistemic status: mostly me noticing (ha) a common pattern among advice i've heard and my personal experience.]

There is one skill or category of skill that is probably worth more than every other general skill of instrumental rationality on this website combined: Noticing things.

You probably already know about noticing confusion, noticing whenever some hypothesis feels "off" or seems inconsistent with your previous beliefs. You probably also already know to notice whenever you're rationalizing or engaging in motivated reasoning.

The remarkable part is that neither of these skills requires you to have a high IQ or learn a lot of new object-level knowledge. At the very least, high IQ or object-level knowledge is not sufficient for these skills. You only need to listen to what your brain is already telling you.

Noticing isn't just for identifying when your thinking is bad, you can also use it to identify when your thinking is good.
Noticing your thoughts or processes of moving from thought to thought allows you to pat yourself on the back and update your cognition or notice which ones aren't doing good work.
Noticing your desires helps you determine whether you should act on them or not. You can act on desires you previously didn't notice, or notice desires you don't want to act on and simply sit and watch them disappear.
Noticing your emotions helps you turn feelings and impulses into rational decisions, or analyze them to figure out their cause.
Babble & Prune follows a similar pattern, where Babble is about noticing ideas, and Prune is about carefully analyzing them, instead of just automatically generating and dismissing them in a single moment.
Noticing is bewelltuned.com's whole deal.
I think most of the benefits of meditation, for me at least, come from it being a way of training your noticing, which incurs the benefits I listed above.
In general, noticing is about taking what you perceive, whether a thought or an emotion or a physical sensation or a subagent or a process or any other mental object, and taking it seriously, not dismissing it out of hand or blindly following it or suppressing it or never having it enter your attention in the first place.
This post is a call to action, not only to notice your everything but also to get people to start talking about noticing and how to notice things better. I'd also like to see what work people have ...
Model Organisms of Misalignment: The Case for a New Pillar of Alignment Research
by evhub — 2023-08-08T01:30:10.847Z

TL;DR: This document lays out the case for research on “model organisms of misalignment” – in vitro demonstrations of the kinds of failures that might pose existential threats – as a new and important pillar of alignment research.

If you’re interested in working on this agenda with us at Anthropic, we’re hiring! Please apply to the

research scientist

or

research engineer

position on the Anthropic website and mention that you’re interested in working on model organisms of misalignment.

The Problem

We don’t currently have ~any strong empirical evidence for the most concerning sources of existential risk, most notably stories around dishonest AI systems that actively trick or fool their training processes or human operators:

Deceptive inner misalignment (a la

Hubinger et al. 2019

): where a model obtains good performance on the training objective, in order to be deployed in the real world and pursue an alternative, misaligned objective.
Sycophantic reward hacking (a la

Cotra 2022

): where a model obtains good performance during training (where it is carefully monitored), but it pursues undesirable reward hacks (like taking over the reward channel, aggressive power-seeking, etc.) during deployment or in domains where it operates with less careful or effective human monitoring.

Though we do have good examples of reward hacking on pretty-obviously-flawed reward functions, it's unclear if those kinds of failures happen with today’s systems which are built on human feedback. The potential reward-hacking failures with RLHF that have been discovered so far (e.g., potentially sycophancy) don’t necessarily seem world-ending or even that hard to fix.

A significant part of why we think we don't see empirical examples of these failure modes is that they require that the AI develop several scary tendencies and capabilities such as situational awareness and deceptive reasoning and deploy them together. As a result, it seems useful to evaluate the likelihood of each of the scary tendencies or capabilities separately in order to understand how severe the risks are from different potential forms of misalignment.

The Plan

We need to develop examples of the above alignment failures, in order to:

Learn more about the possible failures, to understand how likely they are, what causes them to arise, and what techniques may mitigate the failures (discussed in section "The

...
Perpetually Declining Population?
by jefftk — 2023-08-08T01:30:00.897Z

In

With
a Whimper: Depopulation and Longtermism, Geruso and Spears give
the following argument for why most people who'll ever live may have
already died:

People are generally having children below replacement rate: 1.66
children per woman in the US, and total global annual births peaked in
2014.

If you project this forward 300-600 years, annual births fall
below ~10M.

This would leave us with a global population around 560M.

Only a minor disaster could be enough to wipe out humanity once
our population is so low.

They include a pretty bold chart:

To be fair, pretty much any continuation of that chart into the future
is wild,
but the one they've ended up with seems especially so!

I don't find this argument very convincing for several reasons, but I
want to focus on a specific one: even granting all their assumptions I
think we'd see evolution for higher fertility long before we got down
to 10M annual births.

The paper says:

But what, you might ask, about heritability (intergenerational
transmission of high-fertility cultural practices)? Won't the Amish or
some other high-fertility, perhaps religious, sub-population expand to
be as many as we need? For several reasons, no. We have addressed
this question at more length in Arenberg (2022). In the very long
run (i.e., potentially after the coming few centuries of decline), two
facts would have to be true for heritability to be a solution: First,
fertility in a high-fertility sub-group would have to be high enough
(certainly above two, for example). We've already seen above that the
"high fertility" of high fertility subgroups has been declining over
the decades. High fertility used to mean 6 children per woman. Now it
means 2.5. Before long, it may mean 1.8. Second, the children of
high-fertility parents would have to be very likely to remain in their
high-fertility cultural group. Where researchers have studied the
empirical magnitude of these intergenerational correlations as they
have played out in actual practice, they have found them to be
positive, but small—too small, in fact, for the high fertility group
to make much of a dent in overall population. It turns out your
kids might choose not to inherit your cultural practices and beliefs.

If cultural evolution isn't enough, what about genetic evolution?
They cite their 2022 research note Intergeneratio...
A plea for more funding shortfall transparency
by porby — 2023-08-07T21:33:11.912Z

[This post is largely from the perspective of AI safety, but most of it should generalize.]

For recipients, well calibrated estimates about funding probability and quantity are extremely valuable. Funding-dependent individuals and organizations need information to optimize their decisionmaking; incorrect estimates cause waste.

At the moment, getting that information seems unnecessarily hard.

To help with this, I would ask organizations up the funding chain to systematically and continuously provide bite-sized updates from their own perspectives on the funding situation when possible.

This needn't be in the form of a lengthy report or deep-dive (though those are nice too!). For example, for a grantmaking organization with open applications, maybe something like:

We've received V requests for funding totaling $W in the last month. We anticipate funding up to $X of these requests; we would fund up to about $Y if we had more funding.

We don't anticipate significant changes in our funding capacity by default.

Reports like this are already done occasionally. For example, I deeply appreciate reports like this one.[1] My concern is that I'm not aware of any consistent source for this information.

I worry that this is partially because writing a dense report takes a lot of time, and many of these organizations are massively overwhelmed as it is. To the extent that this is limiting reports, I would suggest that giving a tweet-sized miniupdate as things change (or just every few months) would be a huge improvement over the status quo.

Even "oof, we're REALLY funding constrained" would be great! If you don't have time for collecting any numbers at all, a vibecheck is still useful!

There's also no need for each report to attempt to capture the entire field's status, just the perspective of that one organization.[2]

An anecdote of awareness not propagating like it should

For the last six-ish months, I've been trying to figure out how to prioritize earning to give and direct alignment research. I've gotten in touch with several people and heard a number of different perspectives.

None of them were "yeah, the field's pretty funding constrained right now, there are more people than funds, it's a major bottleneck."

This continued a confusion I had starting with the FTX collapse. While I fully expected the field to be in a crunch immediately following the collapse, the vibe I col...
Tips for reducing thinking branching factor
by Simon Berens — 2023-08-07T20:21:43.298Z

Something I notice when I tackle problems of medium+ complexity (top of mind for me is large codebase refactors) my brain tries to explore every possibility in the solution space — every thought generates many more trains of thought to explore, leaving me with decision paralysis.

One solution I’ve been exploring is forcing myself to write down my thought process, but it hasn’t been a resounding success possibly due to high friction.

Has anyone experienced similar problems and have any tips for solving it?
An interactive introduction to grokking and mechanistic interpretability
by Adam Pearce — 2023-08-07T19:09:19.422Z
Link: https://pair.withgoogle.com/explorables/grokking/

Our write up largely agrees with @Quintin Pope's summary, with the addition of training trajectory visualizations and an explanation of the MLP construction that solves modular addition.

A meta note that didn't make it into the article — with so many people looking into this problem over the last 18 months, I'm surprised this construction took so long to find. The modular addition task with a 1-layer MLP is about as simple as you can get![1]

Scaling mechanistic interpretability up to more complex tasks/models seems worth continuing to try, but I'm less sure extracting crisp explanations will be possible.[2] Even if we "solve" superposition, figuring the construction here — where there's no superposition in the generalizing model — wasn't trivial.

gif/twitter summary

If we train a MLP to solve modular addition, the generalizing phase has suggestive periodic patterns.

To figure out why the model generalizes, we first look at task where we know the generalizing solution — sparse parity. You can see the model generalizing as weight decay prunes spurious connections.

One point from the Omnigrok paper I hadn't internalized before training lots of models: grokking only happens when hyper-parameters are just right.

We can make other weird things happen too, like AdamW oscillating between low train loss and low weights.

To understand how a MLP solves modular addition, we train a much smaller model with a circular input embedding baked in.

Following @Neel Nanda and applying a discrete Fourier transform, we see larger models trained from scratch use the same star trick!

Finally, we show what the stars are doing and prove that they work:

Our ReLU activation has a small error, but it's close enough to the exact solution — an x² activation suggested in Grokking modular arithmetic — for the model to patch everything up w/ constructive interference.

^

And there are still open question: why are the frequencies with >5 neurons lopsided? Why does factoring Winput not do that same thing as factoring Woutput?

^

Also see The Hydra Effect: Emergent Self-repair in Language Model Computations