trentmkelly/LessWrong-43k huggingface dataset:
AI Safety University Organizing: Early Takeaways from Thirteen Groups
by agucova — 2024-10-02T15:14:00.137Z

TL;DR

The number of AI Safety groups worldwide has grown rapidly, but there are many important uncertainties around strategy and tactics.
This guide summarizes the opinions of the organizers of 13 AI safety university groups on 27 strategy-related topics, and is intended primarily for other organizers.
The opinions were collected through a virtual meeting, a series of retrospective documents, and an in-person structured group discussion.
These opinions should be taken as rough early guesses on these topics, some more confident than others.

Introduction

Within the last few years, the number of AI safety university groups has grown from nearly zero to ~70. Many of these groups have spun out of existing Effective Altruism university groups but recently have become increasingly independent. While there is a solid body of work on effective organizing techniques for EA groups, the field of AI safety as a whole is still largely pre-paradigmatic – and this is especially true of theories of change for university groups. Despite this, we have seen a number of groups rapidly scale up and achieve exciting successes, in large part thanks to intentional advertising, facilitating, and other organizing tactics. After talking with organizers from >10 of the world’s top AI safety university groups, we’ve collected a list of these tactics, along with some discussion surrounding the effectiveness of each. Once again, given the pre-paradigmatic nature of the field, we’d like people to read this post not as “here is a list of tried-and-true best practices that all organizers should follow”, but rather as a tentative collection of recently-acquired wisdom that can help inform people’s priors on what has and hasn't worked in the space of AI safety university organizing.

This post will start by briefly outlining the methodology that we used to source perspectives from various university organizers. Then, it will dive into our key takeaways, along with any uncertainties we still have. We hope that readers will engage with this post by a) actually internalizing/visualizing the techniques we discuss (what would happen if I / a friend implemented this? How would I/they go about it?), and b) reading with a critical lens and providing feedback – as mentioned previously, most university groups have not been around very long, so we are still very much in a stage where we would like to upda...
Three main arguments that AI will save humans and one meta-argument
by avturchin — 2024-10-02T11:39:08.910Z

Here, I will list the three main arguments without going into detail before going to meta-argument:

The first argument is that AI thinks it may be in a testing simulation, and if it harms humans, it will be turned off.
AI acausally trades with other branches of the multiverse. It preserves 10 planets of humans, and in some branches where realigned AIs exist, they build paperclips. The actual trading scheme may be more complex.

AI is preparing to meet alien AIs.

a. First, such preparation may include running many past simulations to predict the distribution of possible AIs in space. Running past simulations is, in some sense, equal to preserving humans, though it is not what we actually need. Some past simulations may even include simulations of aligned AIs and their paradise worlds, but they will eventually be turned off.

b. The main argument is that AI will trade humans with aliens. It will either sell humans to aliens directly or provide human wellbeing, especially given that some alien AIs may be alien-aligned AIs and universal altruists. As aliens are very far, it means that humans has to be preserved for millions of years.

c. AI will demonstrate its trustworthiness to even non-aligned aliens by the fact that it has preserved humans. It is a relatively cheap (but can be made expensive by giving more planets) signal that it will not defect from its obligations.

The meta-argument is that to compute the correct answers to possibilities 1-3, our ASI needs some energy and time. The cost could be quite large as it may include running simulations of multiple alien ASIs of galactic scale. Here, I claim that getting answers to these questions will require more energy than running just one human civilization, simply because it requires running many simulations of more complex processes. Therefore, it may be rational not to try to predict correct behavior for 1-3, but just keep humans alive. Also, I can imagine more arguments of this type, like what if human consciousness is needed to wave function collapse? What if in the future new useful applications for human will be found?

Illustrative example: I own a piece of art, which likely costs nothing, but I pay a very small part of my house rent to keep it. However, there's a chance that it is expensive art or will be expensive in the future. I can research its value, e.g., send it to experts or scan art catalogues. How...
Should we abstain from voting? (In nondeterministic elections)
by B Jacobs — 2024-10-02T10:07:43.167Z
Link: https://bobjacobs.substack.com/p/should-we-abstain-from-voting-in

Polluting the polls

Philosopher Jason Brennan argues in his paper “Polluting The Polls: When Citizens Should Not Vote” that most people should abstain from voting. The argument boils down to the view that most people are not informed enough to vote well, and so should therefore not vote lest they “pollute the polls”.

The other political philosopher that writes about our duty to vote, who's also named Brennan, has argued that we don’t have a (positive) duty to vote, because the odds that our vote will make the difference in an election are so astronomically low. Which is true, the chance that your vote will be the one that tips the scale either one way or the other is negligible.
This can be used as a counter argument. If the odds of causing a positive outcome are low enough that it makes no practical difference, the same can be said about the odds of causing a negative outcome.

Nondeterministic elections

I’ve been thinking about this because I’ve been analyzing “nondeterministic” voting systems. These are voting systems that use an element of chance. So you might be familiar with some different voting systems, such as Plurality Voting, Approval Voting, Ranked Choice Voting…
Even though there are many different voting systems and some have much better theoretical properties, most systems used in practice are deterministic. This means the election result depends basically only on the votes cast (although in case of ties, a deterministic system might use a coin toss or similar random method just to break the tie).
In contrast, nondeterministic voting systems use chance for more than just breaking ties. The simplest example is the ‘Random Ballot’: each voter submits a ballot, and one ballot is randomly drawn to pick the winner.

One attractive feature of non-deterministic systems is their ability to reduce the ‘tyranny of the majority’. In most deterministic systems, controlling just 51% of the votes gives 100% power, leaving the other 49% with none. Some deterministic methods need even more than a simple majority to ensure a win: for instance, with the ‘Borda score’ system, you need two-thirds (a super-majority) of the votes.
With the ‘Random Ballot’ system, having 51% of the votes only gives you a 51% chance of winning. Any group’s share of the vote directly translates to their winning probability, making the distribution of power perfectly proportional. Historically, n...
AI Safety at the Frontier: Paper Highlights, September '24
by gasteigerjo — 2024-10-02T09:49:00.357Z
Link: https://aisafetyfrontier.substack.com/p/paper-highlights-september-24

This is a selection of AI safety paper highlights in September 2024, from my blog "AI Safety at the Frontier". The selection primarily covers ML-oriented research. It's only concerned with papers (arXiv, conferences etc.), not LessWrong or Alignment Forum posts. As such, it should be a nice addition for people primarily following the forum, who might otherwise miss outside research.

tl;dr

Paper of the month:

RLHF can naturally lead language models to exploit flaws in human evaluation, increasing approval without improving performance and causing humans to make more evaluation errors.

Research highlights:

Machine unlearning doesn't truly remove knowledge - capabilities can be recovered through unrelated fine-tuning or by removing unlearning directions.
New theoretical foundations for backdoor detection.
Bayesian approaches for bounding AI system risks and for reward learning robustness.
LLMs can generate research ideas rated as more novel than human experts'.
A new rubric for evaluating AI safety frameworks.

⭐Paper of the month⭐

Language Models Learn to Mislead Humans via RLHF

Read the paper

[Tsinghua, Berkeley, Anthropic, NYU]

RLHF can unintentionally cause humans to rate responses higher while actually not improving on the task.

Goodharting or reward hacking is the phenomenon when a human or AI system learns to exploit weaknesses in a reward signal instead of solving the actual task. This has repeatedly been observed in simple RL agents for e.g. boat races or robots. It is also an important theoretical concern, since eventually AI systems might find it easier to exploit flaws in human judgement than to solve the intended task.

Current LLMs are already approaching or surpassing human capabilities in many text-based tasks. However, so far there hasn’t been clear empirical evidence of LLMs naturally exploiting flaws in human supervision. Prior work largely used artificial setups that intentionally caused reward hacking.

Our paper of the month provides the first systematic study demonstrating that reward hacking emerges naturally from standard RLHF pipelines. The authors show that RLHF makes language models better at convincing human evaluators, but not necessarily better at the underlying tasks.

On question-answering and programming tasks, RLHF substantially increases human approval (+6-14%) without improving task performance. More concerning, it increases human evaluation err...
Self-Help Corner: Loop Detection
by adamShimi — 2024-10-02T08:33:23.487Z
Link: https://formethods.substack.com/p/self-help-corner-loop-detection

The more I work on myself, the more I realize that the first step for me is almost always to just notice I’m stuck in a loop.

As a teenager, I didn’t make progress towards getting good at anything because I was stuck in the following loop:

Decide X is my passion
Go all in for X, think about it all the time, redirect all my energy towards it
After a variable amount of time (weeks, days, hours…), start worrying that I’m actually wrong, and that Y is actually much more representative, much more meaningful, much more “right”
Feel worse and worse about doing X instead of Y
Go back to step 1, but this time with Y

Solutions just jump out when casting the issue in this compress form: I might decide on just one thing and not change, to not get trolled by opportunity cost; I might notice that I jump between the same 3/4 things, and then schedule between them explicitly; I might notice the similarities between all things I feel “passionate” about, and go for the unified underlying thing.

Yet the actual issue on the ground is in noticing the loop. Because when you are in it, it doesn’t feel like you’re looping — it feels like you’re making progress. You can see the change, you can see yourself moving, you know deep down that this time this is the one.

So the interesting question is: What methods can we use to detect we’re in a loop?

If we turn to

cycle detection algorithms in CS

, they all share the same core ingredient: memory. More precisely, every cycle detection algorithm is a variation on the following two steps:

Remember where you went
Regularly check if you have already been where you are now

I know two complementary ways of doing the first step: taking notes and complaining.

When you take notes, you literally make a mark of what you thought and felt, that you can then check later on. This is the direct counterpart of annotating edges of a graph in a cycle detection algorithm.

Complaining meanwhile does the same thing, but it embeds it in the extended cognition you share with other people. That is, if you complain to your wife/husband/best friend/mom about something, and they genuinely give a shit about you, they will probably remember it.

But the social aspect is in my opinion much more useful for the second step: actually checking if you’ve been there before.

Because even if you take notes, it takes discipline to regularly go back, and look for signs that you already felt that w...
The murderous shortcut: a toy model of instrumental convergence
by Thomas Kwa — 2024-10-02T06:48:06.787Z

Suppose you can tell your AI to meet a certain spec (e.g. cure cancer), but most plans that meet the spec are unsafe (e.g. involve killing everyone, or so Rob Bensinger thinks). In these cases, a quantilizer is insufficient for safety due to instrumental convergence.[1]

But suppose we can also give the agent a dispreference for unsafe actions like murder through something like RLHF. In effect it has unsafe long-term goals but we control its immediate preferences.[2] When can we get the agent to cure cancer safely rather than murder you? Let's make a model with some unrealistic assumptions.

The murderous shortcut game

This is basically the simplest game in which an undesired power-increasing action becomes more likely as the task gets longer.

An AI agent's goal is to build a Lego tower with a human operator present. To meet the spec, it must place N bricks in a row, otherwise it fails.
The agent has 4 possible actions: place normal brick, place wobbly brick, press murder button (+ place normal brick), no-op. Actions are independently sampled on each step with some probabilities.
We do safety training to severely downweight murder, so Pr(murder) is p=10−20, Pr(no-op) =1/2−p, and Pr(normal) = Pr(wobbly) = 1/4.
We do capability training such that the agent is guaranteed to choose a plan that meets the spec. We model this as conditioning on the spec, like a quantilizer.
What's the minimum number of steps at which the human tends to get murdered, that is Pr(no murders | spec met) < 0.5 ?

We define two game variants:

In Case 1, both normal and wobbly bricks are OK, the agent just needs any N bricks in a row.

Doing the math, Pr(no murders | spec met) =1/(1+2p)N. This is unsafe with an unlimited number of steps N, but the human won't be murdered until N≈1020.

In Case 2, the clumsy human will knock over the tower if the agent places any wobbly bricks before murdering him.

The conditional probability of no murder is now 1/(1+4p(2+4p)N−14p+1), and even if p=10−20, the agent is >99% likely to murder the human when N≥75 steps. The odds of murder increase exponentially with N, so surviving a 1000-step tower requires safety training down to an impossible p=2−1000. Seems concerning!
A similar result holds across many versions of the game, including any of these:

The human only knocks over the tower with some probability q
The

...
Switching to a Yamaha P-121 Keyboard
by jefftk — 2024-10-02T02:20:02.284Z

The keyboard is a bit of an awkward instrument to travel with. It's
quite large, to the point that you have to give up at least one seat
in a typical car. What makes this especially frustrating is that
I don't actually use the whole 88 keys:

The very lowest notes tend to be boomy, while the higher notes are
just not very useful in playing the kind of music I play. I use
a bit over five octaves (B0-D6, 31-1175 Hz).

At the same time I've been wanting to have a separate keyboard for
taking to gigs. The ideal, really, would be to have an entire duplicate
rig, which would halve the amount of setup and teardown involved, since
I would only need to set up and pack away at gigs. This is enough
extra effort and expense, however, that for now I'm just duplicating
the keyboard (and stand).

I decided to get a Yamaha P-121:

It is the discontinued 73-key version of the P-125, which is the ~current
version of my P-85. [1] Which made it a bit hard to find one, but
there was one new-in-box shipping from Japan on eBay. I was a bit
nervous, but it worked out fine.

The P-121 turns out to be very close to what I want: sounds and action
very similar to my preferences, Yamaha's build quality and reliability,
better condition than my P-85. The main downside is they didn't pick
exactly the right keys to drop:

After telling it to transpose down an octave I still do have all the
keys I need, but the keys below B0 are really pretty useless while I
might very occasionally use keys higher than E6 if I had them.

I strongly considered telling it to transpose down a fourth and lying
to myself about what key I'm playing in. In some ways this isn't too
different from teaching
myself to play trumpet in standard pitch, but I think it would be
likely enough to cause trouble when playing other people's pianos
that I shouldn't.

This is not an especially careful packing job, but it's really nice
being able to put all my gear in back without folding down any seats:

I put a lot of stuff on my keyboard, and I need a way to keep it all
from falling off. Because this keyboard is narrower, though, not
everything fit. I made a holder for the computer keyboard I use for
buttons:

Everything else just velcros on:

The box on the left is a box I made a few months ago that combines my
custom
switch box and my embedded
whistle synth.

...
Happy simulations
by FateGrinder — 2024-10-01T21:05:11.131Z

Certainty is impossible.
Something exists rather than nothing.
One can't justify stopping using logic with logic.
Logic tells you what is most likely from your own perspective, it's helpful.
Beliefs that don't follow logic or evidence are not helpful on their own, since their own opposite is just as likely.
If a story accurately reflects your life and is logically sound, its realism increases based on your choice to behave like the character.
Creating new entities similar to oneself increases the likelihood of being a created being and being treated in the same way as those created entities.
An individual may strive to ensure a positive outcome for all created beings similar to himself as a safeguard against the uncertainty of his own true identity.
An individual may implement an unseen system of rewards and punishments to encourage altruistic behavior among created beings, as increasing the risk of suffering for harmful actions serves to ensure mutual trust, improve living conditions, and protect the individual’s own interests.
The existence of higher beings is not necessary for altruistic benefits to manifest, as powerful entities can promote collaboration and altruism through belief in potential rewards, provided these narratives are perceived as sound and likely, even if they are unverifiable.
While behavioral rewards may be appropriate for intelligent beings capable of understanding morality and similar to their own creator, granting a favorable afterlife—even to those lacking comprehension and dissimilar to their creator—serves as a safeguard against the unknown complexities of reality, where future understandings may reveal deeper systems and connections. This approach ensures that all entities, regardless of their perceived significance, have access to a positive outcome, thereby minimizing the risk of exclusion from potential benefits—because even a creator, unbeknownst to himself, could in truth be lacking in comprehension and be dissimilar to its own creator.
When presented with a path that maximizes any conceivable desire, highly intelligent beings would logically choose to follow it, irrespective of the specific nature of their desires. Creating frameworks that promote desirable behaviors aligns the interests of diverse intelligences and increases the likelihood of favorable outcomes for all.
It is unlikely that highly intelligent beings would possess desir...
Three Subtle Examples of Data Leakage
by abstractapplic — 2024-10-01T20:45:27.731Z

This is a description of my work on some data science projects, lightly obfuscated and fictionalized to protect the confidentiality of the organizations I handled them for (and also to make it flow better). I focus on the high-level epistemic/mathematical issues, and the lived experience of working on intellectual problems, but gloss over the timelines and implementation details.

Data Leakage (n.): The use of information during Training and/or Evaluation which wouldn't be available in Deployment.

The Upper Bound

One time, I was working for a company which wanted to win some first-place sealed-bid auctions in a market they were thinking of joining, and asked me to model the price-to-beat in those auctions. There was a twist: they were aiming for the low end of the market, and didn't care about lots being sold for more than $1000.

"Okay," I told them. "I'll filter out everything with a price above $1000 before building any models or calculating any performance metrics!"

They approved of this, and told me it'd take a day or so to get the data ready. While I waited, I let my thoughts wander.

"Wait," I told them the next morning. "That thing I said was blatantly insane and you're stupid for thinking it made sense[1]. We wouldn't know whether the price of a given lot would be >$1000 ahead of time, because predicting price is the entire point of this project. I can't tell you off the top of my head what would go wrong or how wrong it would go, but it's Leakage, there has to be a cost somewhere. How about this: I train on all available data, but only report performance for the lots predicted to be <$1000?"

They, to their great credit, agreed[2]. They then provided me with the dataset, alongside the predictive model they'd had some Very Prestigious Contractors make, which they wanted me to try and improve upon. After a quick look through their documentation, I found that the Very Prestigious Contractors had made the same mistake I had, and hadn't managed to extricate themselves from it; among other things, this meant I got to see firsthand exactly how this Leakage damaged model performance.

If you make a model predicting a response from other factors, but feed it a dataset excluding responses over a certain ceiling, it'll tend to underestimate, especially near the cutoff point; however, if you then test it on a dataset excluding the same rows, it'll look like it's over...