trentmkelly/LessWrong-43k huggingface dataset:
Trying to Do More Good
by jefftk — 2024-04-04T14:20:06.303Z

This is an edited transcript of a talk I gave last week at Commonwealth School, a high
school in Boston that I attended from 2000 to 2004. I'm typing from
memory, so in places it may be closer to what I intended to say than
what I actually said.

It's been twenty years since I was a student here, but the place feels
very similar. It's good to be back!

I want to start with a conversation, sometime around 2003. I was in
the computer lab:

That's not actually the computer lab. Would you believe I looked
through four yearbooks and didn't find a single picture of the
computer lab?

Anyway, I was talking to a classmate, in front of some computers that
looked a lot like that one, and we were having some sort of political
or moral argument. They pointed out that what I was arguing people
should do didn't match how I lived, or what I was planning to do with
my life. This felt like a very dirty trick! They were cheating!
Debate is an abstract combat of ideas, it's not supposed to connect to
real life, is it?

I spent the next four years not thinking about this tension very much,
and when I did I felt uneasy and moved on quickly. But then I met my now-wife, who had come to a pretty
different conclusion.
She pushed me pretty hard: how could I live a life that was so far
from my ideals? How could I justify keeping so much for ourselves
when some have so little?

I started thinking: what if I really took this seriously? I talked to others
online,
and the ideas bouncing around this nascent community coalesced into
the effective altruism movement.

While in the movement there were and continue to be lots of different
ideas about what good is and how to do more good, I think at the
core that ties this group together is the idea of comparing different
options and taking optimization seriously.

Over the years I've taken a lot of different approaches to putting
effective altruism into practice. Each time I've decided what to do,
I've had two main questions:

What needs doing?

What's a good fit for me?

Initially there was very little money, including for projects that I
thought were extremely valuable. Perhaps I could earn money and fund
these projects?

We called this "earning to give" which is not a new idea. Sometime
around 1750, John Wesley, the founder of Methodism, advised his
followers to:

Gain all you can

Save

...
Language and Capabilities: Testing LLM Mathematical Abilities Across Languages
by Ethan Edwards — 2024-04-04T13:18:54.909Z

Produced as part of the

SERI ML Alignment Theory Scholars Program

- Summer 2023 Cohort

Thanks to @NicholasKees for guiding this and reading drafts, to @Egg Syntax @Oskar Hollinsworth, @Quentin FEUILLADE--MONTIXI and others for comments and helpful guidance, to @Viktor Rehnberg, @Daniel Paleka, @Sami Petersen, @Leon Lang, Leo Dana, Jacek Karwowski and many others for translation help, and to @ophira for inspiring the original idea.

TLDR

I wanted to test whether GPT-4’s capabilities were dependent on particular idioms, such as the English language and Arabic numerals, or if it could easily translate between languages and numeral systems as a background process. To do this I tested GPT-4’s ability to do 3-digit multiplication problems without Chain-of-Thought using a variety of prompts which included instructions in various natural languages and numeral systems.

Processed results indicated that when properly prompted, GPT is relatively agnostic to the language used and is best at getting correct answers in Arabic numerals, is competent at others including Devangari, Bengali, and Persian, but does rather poorly at less-used numeral systems such as Burmese and Thai. These results are not a function of token-length or GPT not understanding the numeral system.

Performance is also improved by several unexpected factors. Accuracy increases significantly when a restatement of the question (operands and language indicating multiplication) appears in the completion, not just in the prompt. Certain language/numeral-system combinations also trigger completion formulas which have performance boosting factors such as question restatement and Arabic numerals without any clear relation to the instructions in the prompt. Prompting naively in different numeral systems can trigger such performance-boosting conditions in ways that are hard to predict and are not learned by the model by default.

GPT-4 can do math because it has learned particular patterns associated with tokens, including heuristics for certain digits, without fully learning the abstract generalized pattern. These particular token patterns are deserving of more detailed study. GPT-4 is not automatically doing simple processes such as digit-conversion between numeral systems and is relying on the actual content it produces in the completion.

More mechanistic investigations are important for fully underst...
AI #58: Stargate AGI
by Zvi — 2024-04-04T13:10:06.342Z

Another round? Of economists projecting absurdly small impacts, of Google publishing highly valuable research, a cycle of rhetoric, more jailbreaks, and so on. Another great podcast from Dwarkesh Patel, this time going more technical. Another proposed project with a name that reveals quite a lot. A few genuinely new things, as well. On the new offerings front, DALLE-3 now allows image editing, so that’s pretty cool.

Table of Contents

Don’t miss out on Dwarkesh Patel’s podcast with Sholto Douglas and Trenton Bricken, which got the full write-up treatment.

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. Never stop learning.

Language Models Don’t Offer Mundane Utility. The internet is still for porn.

Clauding Along. Good at summarization but not fact checking.

Fun With Image Generation. DALLE-3 now has image editing.

Deepfaketown and Botpocalypse Soon. OpenAI previews voice duplication.

They Took Our Jobs. Employment keeps rising, will continue until it goes down.

The Art of the Jailbreak. It’s easy if you try and try again.

Cybersecurity. Things worked out this time.

Get Involved. Technical AI Safety Conference in Tokyo tomorrow.

Introducing. Grok 1.5, 25 YC company models and ‘Dark Gemini.’

In Other AI News. Seriously, Google, stop publishing all your trade secrets.

Stargate AGI. New giant data center project, great choice of cautionary title.

Larry Summers Watch. Economists continue to have faith in nothing happening.

Quiet Speculations. What about interest rates? Also AI personhood.

AI Doomer Dark Money Astroturf Update. OpenPhil annual report.

The Quest for Sane Regulations. The devil is in the details.

The Week in Audio. A few additional offerings this week.

Rhetorical Innovation. The search for better critics continues.

Aligning a Smarter Than Human Intelligence is Difficult. What are human values?

People Are Worried About AI Killing Everyone. Can one man fight the future?

The Lighter Side. The art must have an end other than itself.

Language Models Offer Mundane Utility

A good encapsulation of a common theme here:

Paul Graham: AI will magnify the already great difference in knowledge between the people who are eager to learn and those who aren’t.

If you want to learn, AI will be great at helping you learn.

If you want to avoid learning? ...
Cult of equilibrium
by Templarrr — 2024-04-04T09:19:51.492Z

TLDR: "Solve for the equilibrium" is a nice sentiment, but shouldn't be applied mindlessly, it's not nearly as universal approach as some think.

Longer version:

The phrase "you must solve for the equilibrium" when evaluating something became almost like mantra and a lot of people use it automatically without stopping and thinking.

Do you though? Do you really? Or is it just a fancy way of saying "mind the consequences" (which it shouldn't be)?

"Solve for equilibrium" just tells you what the situation will become if you assume this one action, this one rule, and then let the system achieve equilibrium without additional interventions. When and what real system ever works like this? Those that I encounter daily are mostly "achieve short term results, evaluate medium term, make corrections before equilibrium sets".

The result of the equilibrium solution just tells you the most stable outcome that, when achieved, will try to maintain itself against future interventions. Quite famously in the classic prisoner dilemma the Nash equilibrium result is the worst one for all involved.

There are cases where you want stability, but this isn't nearly universal rule! Modern fighter planes by themselves has the stability of the brick and require thousands micro-corrections from the onboard computers every second. But the same instability grants them incredible maneuvering capabilities, which their more stable older relatives can't pull off, as their "solved for (aerodynamic) equilibrium" air frames would fight against them.

In a chemical sense every steel tool "solved for equilibrium" is a pile of rust. Unless you regularly support it - and even then it's still a pile of rust, just much further down the line. And when that happens - it doesn't make the tool retrospectively useless, you just replace it.

There is nothing wrong in the systems that require constant corrections as long as

it is known that the system is unstable

there are mechanisms in place to make corrections

and resources dedicated to support those mechanism.
Should you refuse this bet in Technicolor Sleeping Beauty?
by Ape in the coat — 2024-04-04T08:55:09.206Z

This is the question for people who didn't read my latest post. Please, try to answer it yourself without spoiling the solution, and then post it in the comments with your reasoning and whether you consider yourself a halfer or a thirder in regular Sleeping Beauty problem.

Technicolor Sleeping Beauty experiment goes mostly as regular Sleeping Beauty experiment:

The participant is put to sleep on Sunday. Then the coin is tossed. If it's Heads the participant will be awakened on Monday. If it's Tails the participant will be awaken both on Monday and on Tuesday, and between these awakenings their memories will be erased. Therefore, while awakening on Tuesday the participants doesn't remember whether they awakened on Monday or not, and they can never be sure on which day (Monday or Tuesday) their current awakening is happening.

The only difference is that in Technicolor, the walls of the room that the participant awakens in changle their color every day from red to blue and vice versa. The initial color is determined randomly: 1/2 that it's red and 1/2 that it's blue.

While awakened during the experiment you are asked whether you would like to take a specific once per experiment bet:

You may bet that the coin is Tails at 2:3 odds. That is: if you bet 300$ and the coin is indeed Tails you win 200$. The bet will be resolved on Wednesday, after the experiment has ended.

You may take this bet only once per experiment and one agreement is enough. If you have two awakenings in this experiment and agreed on any of them - the bet counts as taken. If you agreed on both of them, the bet counts as taken only once.

For reference, in regular Sleeping Beauty problem utility neutral betting odds for once per experiment bet are 1:1, regardless of whether you are a halfer or a thirder, so taking such bet would be a bad idea.
What's with all the bans recently?
by  — 2024-04-04T06:16:49.062Z

Summary: the moderators appear to be soft banning users with 'rate-limits' without feedback. A careful review of each banned user reveals it's common to be banned despite earnestly attempting to contribute to the site. Some of the most intelligent banned users have mainstream instead of EA views on AI.

Note how the punishment lengths are all the same, I think it was a mass ban-wave of 3 week bans:

Gears to ascension was here but is no longer, guess she convinced them it was a mistake.

Have I made any like really dumb or bad comments recently:

https://www.greaterwrong.com/users/gerald-monroe?show=comments

Well I skimmed through it. I don't see anything. Got a healthy margin now on upvotes, thanks April 1.

Over a month ago, I did comment this stinker. Here is what seems to the same take by a very high reputation user here, @Matthew Barnett , on X: https://twitter.com/MatthewJBar/status/1775026007508230199

Must be a pretty common conclusion, and I wanted this site to pick an image that reflects their vision. Like flagpoles with all the world's flags (from coordination to ban AI) and EMS uses cryonics (to give people an alternative to medical ASI).

I asked the moderators:

@habryka says:

I skimmed all comments I made this year, can't find anything that matches to this accusation. What comment did this happen on? Did this happen once or twice or 50 times or...? Any users want to help here, it surely must be obvious.

You can look here: https://www.greaterwrong.com/users/gerald-monroe?show=comments if you want to help me find what habryka could possibly be referring to.

I recall this happening once, Gears called me out on it, and I deleted the comment.

Conditional that this didn't happen this year, why wasn't I informed or punished or something then?

Skimming the currently banned user list:

Let's see why everyone else got banned. Maybe I can infer a pattern from it:

Akram Choudhary : -2 per comment and 1 post at -25. Taking the doomer view here: frankybegs +2.23 karma per comment. This is not bad. Does seem to make comments personal. Decided to enjoy the site and make 16 comments 6-8 days ago. Has some healthy karma on the comments, +6 to +11. That's pretty good by lesswrong standards. No AI views. Ban reason is???

Victor As...
Best in Class Life Improvement
by sapphire — 2024-04-04T01:51:02.556Z

Im gonna retract this one. Recommends some suboptimally dangerous stuff. Also gives people the wrong idea. I have at least tried anything I recced. But I don't personally use all these things. I only ever recommended them to solve problems you actually had. But given how crazy things have gotten in the rat/ea community I no longer really recommend people explore crazy stuff.
What is the purpose and application of AI Debate?
by VojtaKovarik — 2024-04-04T00:38:24.932Z

I think there is an important lack of clarity and shared understanding regarding how people intend to use AI-Safety-via-Debate-style approaches. So I think it would be helpful if there were some people --- who either (i) work on Debate or (ii) believe that Debate is promising --- who could give their answers to the following three questions:

What is, according to you, the purpose of AI Debate?
What problem is it supposed to be solving?
How do you intend AI Debate to be used?
(EG, to generate training data for imitation learning? During deployment, to generate solutions to tasks? During deployment, to check answers produced by other models?)
Do you think that AI Debate is a reasonably promising approach for this?

Disclaimers: Please don't answer 1-2 without also commenting on 3. Also, note that this isn't a "I didn't understand the relevant papers, please explain" question -- I studied those, and still have this question. Further clarifications in a comment.
Concrete empirical research projects in mechanistic anomaly detection
by Erik Jenner — 2024-04-03T23:07:21.502Z

Thanks to Jordan Taylor, Mark Xu, Alex Mallen, and Lawrence Chan for feedback on a draft! This post was mostly written by Erik, but we're all currently collaborating on this research direction.

Mechanistic anomaly detection (MAD) aims to flag when an AI produces outputs for “unusual reasons.” It is similar to mechanistic interpretability but doesn’t demand human understanding. The

Alignment Research Center

(ARC) is trying to formalize “reasons” for an AI’s output using

heuristic arguments

, aiming for an indefinitely scalable solution to MAD.

As a complement to ARC’s theoretical approach, we are excited about empirical research on MAD. Rather than looking for a principled definition of “reasons,” this means creating incrementally harder MAD benchmarks and better MAD methods.

We have been thinking about and working on empirical MAD research for the past months. We believe there are many tractable and useful experiments, only a fraction of which we can run ourselves. This post describes several directions we’re excited about and high-level reasons to work on empirical MAD.

Background: what is mechanistic anomaly detection, and why care?

This post provides a longer introduction to mechanistic anomaly detection. This section recaps that previous post.

In mechanistic anomaly detection, we want to flag when an AI produces an output “for unusual reasons” or “using anomalous mechanisms” relative to what happens on some reference set of inputs. Concretely, a mechanistic anomaly detection task consists of the following components:

A function f that we want to detect anomalies for. In an empirical context, this is likely a neural network.
A distribution of trusted inputs to f. The behavior of f on this distribution is “normal” by assumption.
A distribution of untrusted inputs. We want to classify inputs from this distribution as normal or anomalous (which will depend on f).

To train an anomaly detector for this task, we have access to f and a dataset of trusted inputs. We may also have access to untrusted inputs, but we don’t know which are normal or anomalous during training. This is one of the key challenges for mechanistic anomaly detection—if we had anomaly labels for untrusted data, we would have a supervised learning problem.

There’s currently no generally accepted formal definition of what “anomalous mechanism...
A gentle introduction to mechanistic anomaly detection
by Erik Jenner — 2024-04-03T23:06:16.778Z

TL;DR: Mechanistic anomaly detection aims to flag when an AI produces outputs for “unusual reasons.” It is similar to mechanistic interpretability but doesn’t demand human understanding. I give a self-contained introduction to mechanistic anomaly detection from a slightly different angle than the

existing one by Paul Christiano

(focused less on heuristic arguments and drawing a more explicit parallel to interpretability).

Mechanistic anomaly detection was first introduced by the Alignment Research Center (ARC), and a lot of this post is based on their ideas. However, I am not affiliated with ARC; this post represents my perspective.

Introduction

We want to create useful AI systems that never do anything too bad. Mechanistic anomaly detection relaxes this goal in two big ways:

Instead of eliminating all bad behavior from the start, we’re just aiming to flag AI outputs online.
Instead of specifically flagging bad outputs, we flag any outputs that the AI produced for “unusual reasons.”

These are serious simplifications. But strong methods for mechanistic anomaly detection (or MAD for short) might still be important progress toward the full goal or even achieve it entirely:

Reliably flagging bad behavior would certainly be a meaningful step (and perhaps sufficient if we can use the detector as a training signal or are just fine with discarding some outputs).
Not all the cases flagged as unusual by MAD will be bad, but the hope is that the converse holds: with the right notion of “unusual reasons,” all bad cases might involve unusual reasons. Often we may be fine with flagging more cases than just the bad ones, as long as it’s not excessive.

I intentionally say “unusual reasons for an output” rather than “unusual inputs” or “unusual outputs.” Good and bad outputs could look indistinguishable to us if they are sufficiently complex, and inputs might have similar problems. The focus on mechanistic anomalies (or “unusual reasons”) distinguishes MAD from other out-of-distribution or anomaly detection problems. Because of this, I read the name as “[mechanistic anomaly] detection”—it’s about detecting mechanistic anomalies rather than detecting any anomalies with mechanistic means.

One intuition pump for mechanistic anomaly detection comes from mechanistic interpretability. If we understand an AI system sufficiently well, we should be ...
$250K in Prizes: SafeBench Competition Announcement

by ozhang — 2024-04-03T22:07:41.171Z

TLDR: CAIS is distributing $250,000 in prizes for benchmarks that empirically assess AI safety. This project is supported by Schmidt Sciences, submissions are open until February 25th, 2025. Winners will be announced April 25th, 2025.
To view additional info about the competition, including submission guidelines and FAQs, visit https://www.mlsafety.org/safebench
If you are interested in receiving updates about SafeBench, feel free to sign up on the homepage using the link above.

About the Competition:
The Center for AI Safety is offering prizes for the best benchmarks across the following four categories:

Robustness: designing systems to be reliable in the face of adversaries and highly unusual situations.
Monitoring: detect malicious use, monitor predictions, and discover unexpected model functionality Alignment: building models that represent and safely optimize difficult-to-specify human values.
Safety Applications: using ML to address broader risks related to how ML systems are handled.

Judges:

Zico Kolter, Carnegie Mellon
Mark Greaves, AI2050
Bo Li, University of Chicago
Dan Hendrycks, Center for AI Safety

Timeline:

Mar 25, 2024: Competition Opens

Feb 25, 2025: Submission

Deadline Apr 25, 2025: Winners Announced

Competition Details:

Prizes: There will be three prizes worth $50,000 and five prizes worth $20,000.

Eligibility: Benchmarks released prior to the competition launch are ineligible for prize consideration. Benchmarks released after competition launch are eligible. More details about prize eligibility can be found in our terms and conditions.

Evaluation criteria: Benchmarks will be assessed according to this evaluation criteria. In order to encourage progress in safety, without also encouraging general advances in capabilities, benchmarks must clearly delineate safety and capabilities.

Submission Format: If you have already written a paper on your benchmark, submit that (as long as it was published after the SafeBench launch date of March 25th, 2024). Otherwise, you may submit a thorough write-up of your benchmark, including source code. An example of such a write-up can be found in this document.

Dataset Policy: By default, we will require the code and dataset for all submissions to be publicly available on Github. However, if the submission deals with a dangerous capability, we will review whether to publicly release the dataset on a ca...