trentmkelly/LessWrong-43k huggingface dataset:
Against Nonlinear (Thing Of Things)
by tailcalled — 2024-01-18T21:40:00.369Z
Link: https://thingofthings.substack.com/p/against-nonlinear

Against Nonlinear is a followup that Ozy wrote to Practically A Book Review: Appendix to "Nonlinear's Evidence: Debunking False and Misleading Claims". In it, they lay out 4 trends in Effective Altruism where Nonlinear was especially bad:

Do It For The Gram: Nonlinear was turning Effective Altruism into being Instagrammable instead of making the world better
So-Called Top EAs: Nonlinear was ranking EAs based on their participation in exclusive parties
File Your Paperwork: Nonlinear was disregarding legally mandated paperwork or generally important accounting
Sunshine Is The Best Disinfectant: Nonlinear was exploiting and undermining EA's trustworthiness
Toward A Mathematical Framework for Computation in Superposition
by Dmitry Vaintrob — 2024-01-18T21:06:57.040Z

Author order randomized. Authors contributed roughly equally — see attribution section for details.

Update as of July 2024: we have collaborated with

@LawrenceC

to expand section 1 of this post into an

arXiv paper

, which culminates in a formal proof that computation in superposition can be leveraged to emulate sparse boolean circuits of arbitrary depth in small neural networks.

What kind of document is this?

What you have in front of you is so far a rough writeup rather than a clean text. As we realized that our work is currently highly relevant to recent questions posed by interpretability researchers, we put together a lightly edited version of private notes we've written over the last ~4 months. If you'd be interested in writing up a cleaner version, get in touch, or just do it. We're making these notes public before we're done with the project because of some combination of (1) seeing others think along similar lines and wanting to make it less likely that people (including us) spend time duplicating work, (2) providing a frame which we think provides plenty of concrete immediate problems for people to independently work on[1] (3) seeking feedback to decrease the chance we spend a bunch of time on nonsense.

1 minute summary

Superposition

is a mechanism that might allow neural networks to represent the values of many more features than they have neurons, provided that those features are present sparsely in the dataset. However, until now, an understanding of how computation can be done in a compressed way directly on these stored features has been limited to a few very specific tasks (for example

here

). The goal of this post is to lay the groundwork for a picture of how computation in superposition can be done in general. We hope this will enable future research to build interpretability techniques for reverse engineering circuits that are manifestly in superposition.

Our main contributions are:

Formalisation of some tasks performed by MLPs and attention layers in terms of computation on boolean features stored in superposition.
A family of novel constructions which allow a single layer MLP to compute a large number of boolean functions of features entirely in superposition.
Discussion of how these constructions could be leveraged:

to emulate arbitrary large sparse boolean circuits entirely in superposition
to allow the QK circuit of an attention head to d

...
The True Story of How GPT-2 Became Maximally Lewd
by Writer — 2024-01-18T21:03:08.167Z
Link: https://youtu.be/qV_rOlHjvvs

This video recounts an incident that occurred at OpenAI in which flipping a single minus sign led the RLHF process to make GPT-2 only output sexually explicit continuations.

The incident is described in OpenAI's paper "Fine-Tuning Language Models from Human Preferences" under section 4.4: "Bugs can optimize for bad behavior".

The script has been written by Jai, with some significant input and rework by me, Writer. You can read it below.

In 2019, one OpenAI researcher made a typo - and birthed an evil AI hell-bent on making everything as horny as possible.

This is the absurd, ridiculous, and yet true story of how it happened.

Part I: GPT

Since 2017, OpenAI has been building Generative Pre-trained Transformer models, or GPTs - language AIs with a singular focus on predicting text, trained across billions of writing samples. If you prompt a GPT model with "Once upon a ", it would predict "time" to follow. Asked for further predictions, the same GPT model might continue "there was a… brave dog named Grace", and so on - because those are the kinds of words that it expects to come next. In this example the GPT model has essentially learned to write a fairy tale, simply as a consequence of getting very, very good at text prediction. And it was exactly these kinds of emergent capabilities that had OpenAI so excited. These models can do a lot more than fairy tales.

OpenAI's first GPT model, often called GPT-1, had been trained on excerpts from thousands of books. It showed so much promise that OpenAI almost immediately decided to train a much bigger model that could do more. But bigger models need more training data, and for this model, books would not be enough. No - this model would be trained on...the Internet.

OpenAI trained GPT-2 to imitate writing across eight million web pages. And in learning to predict such an overwhelming quantity and variety of writing, GPT-2 acquired some surprising capabilities. With the right prompt, it could translate documents, answer questions about a text, summarize passages, and sometimes even write like a human. It was a shockingly powerful model.

In fact, it may have been too powerful. GPT-2 wouldn't hesitate to plan crimes, instruct terrorists on bomb-making, create sexually explicit content, or promote cruelty, hatred and misinformation.

And this was unacceptable to OpenAI - They wanted a model that did more than ju...
Gaia Network: An Illustrated Primer
by Rafael Kaufmann Nedal — 2024-01-18T18:23:25.295Z

NB: this post has lost footnotes. Please read the version on Effective Altruism Forum with footnotes.

In our

first LW post

on the Gaia Network, we

framed it as a solution to the challenges of building safe, transformative AI. However, the true potential of Gaia as a “world-wide web of causal models” goes far beyond that, and in fact, justifying it in terms of its value to other use cases is key to showing its viability for AI safety. At the same time, the previous post focused more on the “what” and “why”, and didn’t really talk much about the “how”. In this piece, we’ll correct both of these flaws: we’ll visually walk through the Gaia Network’s mechanics, with concrete use cases in mind.

The first two parts will cover use cases related to making science more effective and efficient. These would already be sufficient to justify the importance of building the Gaia Network: as “

science is the only news

”, improving science can have a huge positive multiplier effect on our future survival and prosperity. Yet despite a workforce of 8.8 million researchers and funding that adds up to 1.7% of global GDP, science is rightly criticized for inefficiency and limited accountability. The third part will expand beyond the epistemic (scientific) benefits of the Gaia Network and towards pragmatic impact - ie, making all decision-making more effective and efficient, which impacts the entire world population and GDP. And the last two sections will focus on the applications of the Gaia Network on existential risk - first specifically with regard to AI safety, and finally as a general tool for collective sensemaking and coordination around the Metacrisis.

For brevity’s sake, we will not cover any of the implementation details or mathematical grounding. We’ll focus on the core concepts and capabilities, and try to explain them in plain language. We’ll also skim over much of the “hard parts”: the economics and trust modeling. Finally, we will not cover the arguments for convergence and resilience of the network; these have been already sketched out in our

previous paper

, and merit a more formal and in-depth analysis than we can incorporate into this primer. If there’s some hand-waving in the below that makes you uncomfortable, please let us know in the comments and we will attempt to assuage you.

The beginning will take a bit long with Bayesian statistics, as thes...
On the abolition of man
by Joe Carlsmith — 2024-01-18T18:17:06.201Z

(Cross-posted from my website. Podcast version here, or search for "Joe Carlsmith Audio" on your podcast app.

This essay is part of a series that I'm calling "Otherness and control
in the age of AGI." I'm hoping that the individual essay can be read
fairly well on their own, but see
here
for brief summaries of the essays that have been released thus far.)

Earlier in this
series,
I discussed a certain kind of concern about the AI alignment discourse – namely, that it aspires to exert an inappropriate degree of control
over the values that guide the future. In considering this concern, I
think it's important to bear in mind the aspects of our own values
that are specifically focused on pluralism, tolerance, helpfulness, and
inclusivity towards values different-from-our-own (I discussed these in
the last
essay).
But I don't think this is enough, on its own, to fully allay the concern
in question. Here I want to analyze one version of this concern more
directly, and to try to understand what an adequate response could
consist in.

Tyrants and poultry-keepers

Have you read The Abolition of
Man,
by C.S. Lewis? As usual: no worries if not (I'll summarize it in a
second). But: recommended. In particular: The Abolition of Man is
written in opposition to something closely akin to the sort of
Yudkowskian worldview and orientation towards the future that I've been
discussing.[1] I think the book is wrong about a bunch of stuff. But I
also think that it's an instructive evocation of a particular way of
being concerned about controlling future values – one that I think
other critics of Yudkowskian vibes (e.g.,
Hanson)
often draw on as well.[2]

At its core, The Abolition of Man is about meta-ethics. Basically,
Lewis thinks that some kind of moral realism is true. In particular, he
thinks cultures and religions worldwide have all rightly recognized
something he calls the Tao – some kind of natural law; a way that
rightly reflects and responds to the world; an ethics that is objective,
authoritative, and deeply tied to the nature of Being itself. Indeed,
Lewis thinks that the content of human morality across cultures and
time periods has been broadly similar, and he includes, in the appendix
of the book, a smattering of quotations meant to illustrate (though not:
establish) this point.

"Laozi Riding an Ox by Zhang
Lu
(c. 1464--1538)" (Image source
here)

But Lewis notices, also, th...
More Usable Recipes
by jefftk — 2024-01-18T17:40:05.292Z

Instead of a recipe book I keep a

web page.
Whenever I make something and think I'll make it again, I add it.
It's public since I generally make things public by default, though
it's terse since it's intended as a reference for someone (me!) who
already knows what to make. Recipes looked something like:

I've now updated them so I can check things off as I go:

Since I author
in HTML, I didn't want to turn my tidy source from:

<li>2 eggs (or 2T flax and 5T water)
<li>2/3 oil
<li>1C greek yoghurt
<li>1/4 cup milk, more if needed

Into something ugly like:

<li><label><input type=checkbox></input>
2 eggs (or 2T flax and 5T water)</label>
<li><label><input type=checkbox></input>
2/3 oil</label>
<li><label><input type=checkbox></input>
1C greek yoghurt</label>
<li><label><input type=checkbox></input>
1/4 cup milk, more if needed</label>

Instead, I've added a little JS to each page that does this at display
time:

for (const li of
document.getElementsByTagName("li")) {
li.innerHTML =
"<label><input type=checkbox></input>" +
li.innerText + "</label>";
}
}

I've also added a bit of CSS for extra vertical space
for the checkbox but only on devices like phones where big sloppy
fingers need it:

<style>
@media (pointer:coarse) {
input {
margin-top: 1em;
margin-bottom: 1em;
}
}
</style>

I haven't actually cooked anything since changing this, but I think
I'll like it.
Good job opportunities for helping with the most important century
by HoldenKarnofsky — 2024-01-18T17:30:03.332Z

Yes, this is my first post in almost a year. I’m no longer prioritizing this blog, but I will still occasionally post something.

I wrote ~2 years ago that it was hard to point to concrete opportunities to help the most important century go well. That’s changing.

There are a good number of jobs available now that are both really promising opportunities to help (in my opinion) and are suitable for people without a lot of pre-existing knowledge of AI risk (or even AI). The jobs are demanding, but unlike many of the job openings that existed a couple of years ago, they are at well-developed organizations and involve relatively clear goals.

So if you’re someone who wants to help, but has been waiting for the right moment, this might be it. (Or not! I’ll probably keep making posts like this as the set of opportunities gets wider.)

Here are the jobs that best fit this description right now, as far as I can tell. The rest of this post will give a bit more detail on how these jobs can help, what skills they require and why these are the ones I listed.

Organization

Location

Jobs

Link

UK AI Safety Institute

London (remote work possible within the UK)

Engineering and frontend roles, cybersecurity roles

Here

AAAS, Horizon Institute for Public Service, Tech Congress

Washington, DC

Fellowships serving as entry points into US policy roles

Here

AI companies: Google DeepMind, OpenAI, Anthropic1

San Francisco and London (with some other offices and remote work options)

Preparedness/Responsible Scaling roles; alignment research roles

Here, here, here, here

Model Evaluation and Threat Research (METR) (fewer roles available)

Berkeley (with remote work options)

Engineering and data roles

Here

Software engineering and development (and related areas) seem especially valuable right now, so think about whether you know folks with those skills who might be interested!

How these help

A lot of these jobs (and the ones I know the most about) would be contributing toward a possible global standards regime for AI: AI systems should be subject to testing to see whether they present major risks, and training/deploying AI should stopped (e.g., by regulation) when it can’t be done safely.

The basic hope i...
Flexibility and the Singularity
by Jonathan Moregård — 2024-01-18T15:29:02.727Z
Link: https://honestliving.substack.com/p/the-importance-of-exploration

Why am I travelling to Spain? Better weather is a plus, but I’ll miss out on all the great things happening back home. Why pay a bunch of money if I’m not sure it’ll be more enjoyable?

I haven’t travelled abroad since COVID hit. I want to travel abroad at some point in my life - culture clashes are a great way to learn more about myself. If I don’t travel now, when will I start?

It’s common to resist doing new things, dragging your feet and coming up with reasons to avoid taking that first step. Doing something for the “first” time is always the hardest - the second time is way easier. Doing a new thing opens up your option space, increasing the range of things you are comfortable doing.

Here’s another example of me increasing my agency in this way:

I live in Uppsala, a city located next to Sweden’s biggest city, Stockholm. I

moved to Uppsala

partially because I want to go to events happening in Stockholm. Great plan, poor execution. The first six months I lived in Uppsala, I didn’t attend a single evening event in Stockholm. Every time an event came up, I felt tired. The road is too long, the event might suck, and shouldn’t I take are of myself and stay home?

One day, I had enough. I decided to break the pattern. I went to an evening event, even though I didn’t feel like doing it. After attending that first event, the next ones were easier. I now visit Stockholm regularly, going to shibari evenings, clown performances and simply hanging out at friend’s places.

Flexibility and the Singularity

It’s common for people to get stuck in their ways as they grow older. Kids treat everything as new, while older people fall back on habit and prior experience. In our ancestral environment, this made perfect sense - build experience as you grow, and execute as you grow older.

In our modern world, this makes less sense. The world of today is not the world of your childhood - the rules are in constant flux. Acting out the patterns you learned during your childhood is no longer adaptive. To succeed in the modern world, it’s important to explore more, try new ways of being and acting, and stay up-to-date with the world around you.

Other people have written about the need to explore more. A popular text from a few years back is the

live player, dead player

article. The same theme pops up in discussions around

exploring vs exploiting

.

Staying Flexible - Practical Steps

Given...
AI #48: Exponentials in Geometry
by Zvi — 2024-01-18T14:20:07.869Z

The biggest event of the week was the Sleeper Agents paper from Anthropic. I expect that to inform our thoughts for a while to come, and to lay foundation for additional work. We also had the first third of the IMO solved at almost gold metal level by DeepMind, discovering that math competition geometry is actually mostly composed of One Weird Trick. I knew that at the time I was doing it, though, and it was still really hard.

As usual, there was also a bunch of other stuff.

Tomorrow the 19th, I am going to be off to San Francisco for the weekend to attend a workshop. That leaves a lot of time for other events and seeing other people, a lot of which remains unfilled. So if you are interested in meeting up or want to invite me to a gathering, especially on Sunday the 21st, drop me a line.

Table of Contents

Items in bold are recommended as relatively higher value than usual.

Items in italics are anti-recommended as only for those with specific interest.

There is also my coverage of the Anthropic Sleeper Agents Paper.

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. Cybersecurity for everyone.

Language Models Don’t Offer Mundane Utility. Adaptation is slow and difficult.

GPT-4 Real This Time. Guess who is attempting to come to the GPT store.

Fun With Image Generation. It now works how people used to think it works?

Copyright Confrontation. Some want to live forever any way they can get it.

Deepfaketown and Botpocalypse Soon. The atom blaster pointing both ways.

They Took Our Jobs. Dentists use AI the way you would expect given incentives.

Get Involved. Fellowships, technical roles, Anthropic, reminder about MIT.

Introducing. AlphaGeometry aces that part of the IMO.

In Other AI News. Is ‘democratic’ AI distinct from ‘social approval consensus AI’?

Quiet Speculations. DPO, education, timelines, clarifying terms.

The Quest for Sane Regulation. ‘No AI Fraud Act’ is written poorly.

The Week in Audio with Sam Altman. Talks to Gates and to Davos.

David Brin Podcast. I went deeper than I should have gone, I see that now.

Rhetorical Innovation. It’s not progress.

Anthropic Paper on Sleeper Agents. An epilogue. Original seems important.

Anthropic Introduces Impossible Mission Force. I accept it.

Aligning a Smarter Than Human Intelligence is Difficult. So is evaluation.

The Belros

...
Worrisome misunderstanding of the core issues with AI transition
by Roman Leventov — 2024-01-18T10:05:30.088Z

This post is triggered by “Generative AI dominates Davos discussions as companies focus on accuracy” (CNBC) and “AI has a trust problem — meet the startups trying to fix it” (Sifted).

It's just remarkable (and worrying) how business leaders and journalists misunderstand the core issues with AI adoption and transition[1]. All they talk about is "accuracy", "correctness", and "proving that AI is actually right"(!). The second piece has a hilarious passage “Cassar says this aspect of AI systems creates a trust issue because it goes against the human instinct to make ‘rule-based’ decisions.”(!)

There are many short- and medium-term applications where this "rule-following and accuracy" framing of the issue is correct, but they are all, by necessity, about automating and greasing bureaucratic procedures and formal compliance with rule books: filing tax forms, checking compliance with the law, etc. But these applications are not intrinsically productive, and on a longer time scale, they may lead to a Jevons effect: the cheaper bureaucratic compliance becomes, the more it is demanded, without actually making coordination, cooperation, and control more reliable and safe.

"Factual accuracy" and hallucinations are the lowest-hanging pieces of context alignment

Taking the viewpoints of information theory[2], philosophy of language, and institutional economics[3], it's not the sophistication of bureaucracies that reduces the cumulative risk exposure and transaction costs of the interaction between humans, AIs, and organisations. Rather, it's building shared reference frames (shared language) for these agents to communicate about their preferences, plans, and risks. The sophistication of bureaucratic procedures sometimes does have this effect (new concepts are invented that increase the expressiveness of communication about preferences, plans, and risks), but this is only an accidental byproduct of this bureaucratisation process. And then, making AIs use language effectively to communicate with humans and each other is not an "accuracy" or "factual correctness" problem, it's the context (meaning, intent, outer) alignment problem.

Indeed, this is the core problem that Perplexity, Copilot, Bard, OpenAI, and other universal RAG helpers are facing: alignment with users’ context, on a hierarchy of timescales: pre-training[4], fine-tuning[5], RAG dataset curation, and online alignme...