trentmkelly/LessWrong-43k huggingface dataset:
Searching for the Root of the Tree of Evil
by Ivan Vendrov — 2024-06-08T17:05:53.950Z
Link: https://nothinghuman.substack.com/p/searching-for-the-root-of-the-tree

“There are a thousand hacking at the branches of evil to one who is striking at the root”

Henry David Thoreau,

Walden

The world is full of problems: Pain, poverty, illness, war, pollution, to pick a few among thousands. Many of us feel like we need to Do Something about these problems. There’s just one problem (sorry): There are so many problems that focusing on one specific problem feels wrong. Why choose that one? How dare you ignore all the other problems? Do those people suffering not matter to you? Can’t you hear their screams?

One response is the effective altruist’s

shut up and multiply

: loosely, find a framework that lets you rank problems against each other, and then work on the most important problems. Yes, war is hell, but solving it is not tractable or neglected. So you buy

anti-malarial bednets

and try your best to ignore the screams.

Another is to try to identify the root cause of all the problems, and work on that. Some popular candidates include

metascience

, “healing our collective trauma”, “awakening from the meaning crisis”, “solving intelligence and using it to solve everything else”.

Sometimes you can justify this on effective altruist grounds, but it’s also very neat psychologically: you don’t have to ignore the screams. Whenever a new problem comes to your attention, you can say “yes, my work is helping with this also”. You can channel that empathy into whatever you were already doing. And afterwards, you can sleep easy, knowing that you helped.

This is awfully convenient, and you might be tempted to dismiss it as cope, as a way to feel good about yourself without ever doing any real, object-level, devil-is-in-the-details kind of work. And yet… this meta-strategy has worked at least once before, and worked so well as to dwarf all the object-level charity ever wrought by human hands and hearts. I’m referring to Francis Bacon’s invention of science.

Quoting the historian and novelist Ada Palmer’s beautiful essay

On Progress and Historical Change

:

Then in the early seventeenth century, Francis Bacon invented progress.

If we work together — said he — if we observe the world around us, study, share our findings, collaborate, uncover as a human team the secret causes of things hidden in nature, we can base new inventions on our new knowledge which will, in small ways, little by little, make human life just a little easier, just a l

...
2. Corrigibility Intuition
by Max Harms — 2024-06-08T15:52:29.971Z

(Part 2 of the CAST sequence)

As a reminder, here’s how I’ve been defining “corrigible” when introducing the concept: an agent is corrigible when it robustly acts opposite of the trope of "be careful what you wish for" by cautiously reflecting on itself as a flawed tool and focusing on empowering the principal to fix its flaws and mistakes.

This definition is vague, imprecise, and hides a lot of nuance. What do we mean by “flaws,” for example? Even the parts that may seem most solid, such as the notion of there being a principal and an agent, may seem philosophically confused to a sufficiently advanced mind. We’ll get into trying to precisely formalize corrigibility later on, but part of the point of corrigibility is to work even when it’s only loosely understood. I’m more interested in looking for something robust (i.e. simple and gravitational) that can be easily gestured at, rather than trying to find something that has a precise, unimpeachable construction.[1]

Towards this end, I think it’s valuable to try and get a rich, intuitive feeling for what I’m trying to talk about, and only attempt technical details once there’s a shared sense of the outline. So in this document I’ll attempt to build up details around what I mean by “corrigibility” through small stories about a purely corrigible agent whom I’ll call Cora, and her principal, who I’ll name Prince. These stories will attempt to demonstrate how some desiderata (such as obedience) emerge naturally from corrigibility, while others (like kindness) do not, as well as provide some texture on the ways in which the plain-English definition above is incomplete. Please keep in mind that these stories are meant to illustrate what we want, rather than how to get what we want; actually producing an agent that actually has all the corrigibility desiderata will take a deeper, better training set than just feeding these stories to a language model or whatever. In the end, corrigibility is not the definition given above, nor is it the collection of these desiderata, but rather corrigibility is the simple concept which generates the desiderata and which might be loosely described by my attempt at a definition.

I’m going to be vague about the nature of Cora in these stories, with an implication that she’s a somewhat humanoid entity with some powers, a bit like a genie. It probably works best if y...
Two easy things that maybe Just Work to improve AI discourse
by Bird Concept — 2024-06-08T15:51:18.078Z

So, it seems AI discourse on X / Twitter is getting polarised. This is bad. Especially bad is how some engage in deliberate weaponization of discourse, for political ends.

At the same time, I observe: AI Twitter is still a small space. There are often important posts that have only ~100 likes, ~10-100 comments, and maybe ~10-30 likes on top comments. Moreover, it seems to me little sane comments, when they do appear, do get upvoted.

This is... crazy! Consider this thread:

A piece of legislation is being discussed, with major ramifications for regulation of frontier models, and... the quality of discourse hinges on whether 5-10 random folks show up and say some sensible stuff on Twitter!?

It took me a while to see these things. I think I had a cached view of "political discourse is hopeless, the masses of trolls are too big for anything to matter, unless you've got some specialised lever or run one of these platforms".

I now think I was wrong.

Just like I was wrong for many years about the feasibility of getting public and regulatory support for taking AI risk seriously.

This begets the following hypothesis: AI discourse might currently be small enough that we could basically just brute force raise the sanity waterline. No galaxy-brained stuff. Just a flood of folks making... reasonable arguments.

It's the dumbest possible plan: let's improve AI discourse by going to places with bad discourse and making good arguments.

I recognise this is a pretty strange view, and does counter a lot of priors I've built up hanging around LessWrong for the last couple years. If it works, it's because of a surprising, contingent, state of affairs. In a few months or years the numbers might shake out differently. But for the time being, the arbitrage is real.

Furthermore, there's of course already a built-in feature, with beautiful mechanism design and strong buy-in from X leadership, for increasing the sanity waterline: Community Notes. It's a feature that allows users to add "notes" to tweets providing context, and then only shows those notes ~if they get upvoted by people who usually disagree.

Yet... outside of massive news like the OpenAI NDA scandal, Community Notes is barely being used for AI discourse. I'd guess probably for no more interesting of a reason than that few people use community notes overall, multiplied by few...
I made an AI safety fellowship. What I wish I knew.
by Ruben Castaing — 2024-06-08T15:23:26.618Z

TLDR

If you make an AI safety course: Leverage other people. Make sure to cover why AI safety is important. Build a community and just start. Use the below resources and slides.

Why Bother

Making an AI safety fellowship is highly worth doing. It has likely been a major update on my and other student's career paths. For approximately 150 hours of work, it has been the highest impact thing I've done all year. I've learnt so much from teaching and have met some great people doing so. We made an 8-week AI safety course using the course materials from Blue Dot. There were around 9 people each week, mostly studying computer science. This is in person in Christchurch New Zealand. We will run next semester.

What I wish I knew:

Leverage other people.

It surprised me just how willing other people were to help us. It's likely because AI safety fellowships are cool and useful. The diverse range of skills is something I could have never replicated myself. For example, our university computer science society helped us advertise to most of the cohort. ANZ AI Safety is giving us marketing advice. Our Effective Altruism group gave us some funding. We have a new co-facilitator next semester. The best way to organize this is by direct messaging people and online or in-person conversations.

Beware the AI buzz

Many people joined because of an interest in AI. They don't know how important AI safety is. I recommend spending the first week discussing why AI safety is important. Blue Dots readings don't spend enough time on this, likely as signups already recognize the importance of AI safety. We will likely do this by summarizing AI safety from first principles.

Build a community

People often stayed within their social groups. This made people less willing to share viewpoints they were unsure of. I should have encouraged a culture of being ok with being wrong. An intro pizza session would have helped.

Just start

I didn't feel ready when I started. I'm glad I did so anyway. You don't have to know everything. You will learn from teaching and can leverage the group's combined knowledge. In the long term, having the initiative to start is what ends up giving you useful skills. Facilitating is fun, which helps you keep going. People will appreciate the effort, so will accept your mistakes.

Course materials

Blue Dot course readings are high quality. I highly recommend showing Andrej Karpat...
Alignment Gaps
by kcyras — 2024-06-08T15:23:16.396Z

Misaligned agendas and terminology among academic, industrial and independent AI alignment research

This post aims to fill some gaps between technical AI Alignment topics and academic AI research.
It summarises a quick but informed scouting of academic research papers that are closely connected to four Alignment topics: mechanistic interpretability, safety via debate, value alignment and cooperative AI. This overview should help somebody wanting to do research on any of those four topics to find the related research and expand the technical Alignment frontier.

Audience: somebody who knows what research is (typically a PhD) and is familiar with technical topics in AI Alignment

Epistemic status: informed guess after going over recent AI Alignment research papers; biased towards academic AI research on formal methods, especially argumentation, but more ignorant about ML

TLDR: mechanistic interpretability can be seen as a form of formal explainability; AI safety via debate is vague and informal and could benefit from formal argumentative dialogue games (and has in fact been formalised using several formalisms); value alignment (including RLHF and similar) has deep connections to computational social choice theory and reasoning with preferences in AI; cooperative AI might benefit from recent fairness and multi-agent research.

Takeaway

My key takeaway is that technical Alignment research pre-prints often fail to refer to peer-reviewed, formal AI research articles that are highly relevant to the Alignment topic studied, but that do not use the typical Alignment terminology. This may be due to several reasons.
For one, some academic research pre-dates Alignment topics -- for instance, formal verification and explainability via formal methods go way back before mechanistic interpretability (or even machine learning (ML) interpretability at large).
In other cases, authors of non-peer-reviewed papers may simply be unaware of, or may intentionally omit related research -- for instance, the AI safety via debate pre-print does not even mention some two decades of research on argumentative debates and dialogue games.
In yet other cases, the authors seem to be aware of some relevant research but fail to explore either deeper or broader connections -- for instance, the authors of What are human values, and how do we align AI to them? acknowledge and use some insights from s...
The Slack Double Crux, or how to negotiate with yourself
by  — 2024-06-08T15:22:58.533Z

This is a post mostly about Slack. Slack in the sense of Zvi, Slack as the freedom to not be bound by an obligation to have to do anything. Slack is generally accepted to be good and worth pursuing. This post is about the phenomenon of actions which increase Slack also tending to decrease Slack, and what to do about that.
This post is also about IFS, because Internal Family Systems theory is, in my opinion, among the best psychotechnology available right now.
This is also my first post, I hope it is to someone's liking.
A very simple example for an action that both decreases and increases slack: Having certain days/hours in which you are not allowed to play video games.
This increases slack, as long as you are addicted, aka got got by videogames, being able to not play them while you desire to, opens up a window in which you are free to do anything else, aka the desirable Slack.
The problem however is that not being allowed to do something also naturally decreases Slack. Maybe playing a video game on this day, on this hour is already the perfect economic decision, something which Slack is supposed to help you achieve. Maybe you are so stressed that you need to decompress, and a video game is the perfect tool at your disposal. Maybe the video game is the artsy type, more Disco Elysium than Fortnite, and playing it will broaden your horizon, give you access to new culture or thought models, enrich your creativity, or do any of the other nice things that good art tends to do, and that option is superior to any of your other options at the time and fits you needs best.

So, it stands to reason, that most actions actually increase and decrease Slack at different rates and in different aspects. Every decision to modify your behaviour will set rules for you to follow, which will decrease Slack, but benefit you by culling negative behaviour, which increases Slack. Abolishing rules does the inverse. So now that the impact on every action on Slack is fuzzy and complicated, we need tools to decide wether the action or rule or behaviour consults in net Slack loss or Slack gain.
The best toolset for this job I have found is IFS. IFS - Internal Family Systems Therapy - is a new type of therapy that is taking the ratsphere by storm. Scott Alexander writes about it, Kaj Sotala writes about it, a bunch of other people write about it. It is hot right now.
I also thin...
Status quo bias is usually justified
by Amadeus Pagel — 2024-06-08T14:54:13.648Z
Link: https://amadeuspagel.substack.com/p/status-quo-bias-is-usually-justified

Generally, we know more about the status quo then about anything else.

We know that we can live in the current climate, we don’t know that about any other climate.
We know that society functions with current laws and norms, we don’t know that about any other set of laws.

Often, we are adapted to the status quo and the status quo is adapted to us.

We and the plants and animals we depend on evolved to live in the current the global climate. Many of our city are built on the coast, our population is distributed in a way that can be supported by agriculture.
Our laws are the result of millenia of trial and error.

Often, the status quo is a compromise between different groups, individuals or even desires within an indivdual.

The test for status quo bias is reversal. If we oppose a change in one direction, do we also oppose change in the other direction? We’re concerned about global warming, and we were also concerned about

global cooling

. Any change in the climate has unpredictable consequences.
Closed-Source Evaluations
by Jono — 2024-06-08T14:18:40.800Z

Public tripwires are no tripwires.

I'm writing a quick and dirty post because the alternative is that I wait for months and maybe not write it after all. I am broadly familiar with the state of interpretability research but do not know what the state of model evaluations is at the moment.

The interpretability win screen comes after the game over.

There is an interpretability-enabled win condition at the point where billion-parameter networks become transparent enough that we robustly can detect deception.
We are paperclipped long before that since their adjacant insights predictably lead to quicker iteration cycles, new architecture discoveries, insight into strategic planning and other capabilities AGI labs are looking for.

Tripwires stop working when you train models to avoid them.

Current interpretability research solely (as far as I'm aware) produces tools that melt under slight optimization pressure. Optimising on an interpretability tool, optimizes against being interpretable.
It would be easy to fool others and oneself into thinking some model is safe because the non-optimisation-resistant interpretability tool showed your model was safe, after the model was optimised on it. If not that, then you could still be fooled into thinking you didn't optimise on the interpretability tool or its ramifications.

You cannot learn by trial and error if you are not allowed to fail.

We could fire rockets at the moon because we could fail at many intermediate points, with AGI, we're no longer allowed to fail after a certain threshold. Continuing to rely on the useful insights from failur,e, is thus a doomed approach[1] and interpretability increases the use AGI labs get out of failure.
To start preparing for a world in which we're not allowed to fail, we should build one where failing hurts instead of helps AGI-creators.

Interpretability should go to closed-source evaluators.

We should close-source interpretability and have trustworthy evaluators buy new tools up and solely use them to evaluate frontier models in a tripwire approach. These evaluators should then not tell AGI labs what failed, just that it failed[2]. Ideally evaluators get to block model deployments of course, but them having a good track record of warning against upcoming model failures[3] is a good start.

AGI labs become incentivised to anticipate failure and lose ability to argue anyone into thi...
Access to powerful AI might make computer security radically easier
by Buck — 2024-06-08T06:00:19.310Z

People talk about model weight security being really hard and crucial around the advent of AGI. (E.g.

RAND report

,

Leopold

; see

here

for some distinctions in these threat models that I think are important.) But I think that the thinking on this has not been sufficiently attentive to the fact that during that crucial time period, by assumption we’ll have access to powerful AIs. I think that such access might make security wildly easier, by a bunch of different mechanisms, some of which I’ll describe in this post.

The story I’m telling here is pretty different from the main story I've heard people talk about in the past about AIs helping with computer security, which is that AIs can help with hardening software and other infrastructure. Though I agree that that seems really useful and important, here I'm not talking about AIs automating the process of security research, I'm talking about AI making it affordable to adopt policies that improve security in the organization but that wouldn’t work (because they’d be too labor-intensive or lead to who-guards-the-guards problems) if you just had humans.

In this post I’m going to be ignoring problems related to AIs themselves trying to cause security failures. There are two ways that such AIs affect this situation:

The techniques I describe in this post could also be used, AI-control style, to use

trusted models

to reduce the risk of security failures caused by untrusted models. You can basically just replace “humans” by “untrusted models” everywhere in the text below.
When applying these techniques to reduce risk from humans, you have to worry that these techniques will be sabotaged by scheming models. I’ll ignore that possibility in this post, and talk about ways that you can improve security using AIs that are assumed not to be intentionally trying to subvert these security measures (which we call “

trusted

”). If you’re worried about your models scheming, you’ll have to do some other stuff too (mostly various sting operations), but I think the arguments in this post still mostly hold up. I’ll talk about this more in future.

I’m going to focus on insider threats here; most threats from outside humans involve first gaining access to impersonation of some insider.

Much of this post came out of thinking about the question “how could

AI control techniques

be applied to humans, instead of AIs”?

I don’t t...
Why don't we just get rid of all the bioethicists?
by Sable — 2024-06-08T03:48:47.760Z

I've had a few vague answers before, but as I was considering how no one was allowed to do human challenge trials during COVID, I thought I'd ask:

Can anyone come up with an example where a bioethicist actually improved the world?

I'm looking for specific cases, not a general "nazi science was horrible" sort of thing. Is there a record of horrifying things that IRBs have prevented?

If, tomorrow, all bioethicists were fired and IRBs basically disappeared, would that unleash a wave of evil scientists who are barely kept in check now, or would everything be completely fine (even better, possibly)?
Sev, Sevteen, Sevty, Sevth
by jefftk — 2024-06-08T02:30:02.040Z

I don't like the number seven. Well, really the name of the number
seven. All the other single digit numbers are single syllable, and
seven has to go and take two. Seventy and seventeen have the same
problem. What can we do?

I think the two main candidates are "sev" (dropping the second
syllable) and "sen" (dropping the first coda and second onset). While
I find "sen" slightly nicer on the tongue, I think "sev" is more
promising because it feels like a better short form.

It feels like we ought to be able to switch to calling it "sev", where
some people just start saying that and other people understand them?
I've been playing around it, but every time I do my toddler Nora
laughs at me as if I'm being ridiculously over the top: "you said
sev!!" Does not bode well for a low-key
migration.

Comment via: facebook, mastodon