trentmkelly/LessWrong-43k huggingface dataset:
Pondering how good or bad things will be in the AGI future
by Sherrinford — 2024-07-09T22:46:31.874Z

Yesterday I heard a podcast where someone said he hoped AGI would be developed in his lifetime. This confused me, and I realized that it might be useful - at least for me - to write down this confusion.

Consider that for some reasons - different history, different natural laws, whatever - LLMs had never been invented, the AI winter had taken forever, and AGI would generally be impossible. Progress would still have been possible in this hypothetical world, but without whatever is called AI nowadays or in the real-world future.

Such a world seems enjoyable. It is plausible that technological and political progress might get it to fulfilling all Sustainable Development Goals. Yes, death would still exist (though people might live much longer than they currently do). Yes, existential risks to humanity would still exist, although they might be smaller and hopefully kept in check. Yes, sadness and other bad feelings would still exist. Mental health would potentially fare very well in the long term (but perhaps poorly in the short term, due to smartphones or whatever). Overall, if I had to choose between living in the 2010s and not living at all, I think the 2010s were the much better choice, as were the 2000s and the 1990s (at least for the average population in my area). And the hypothetical 2010s (or hypothetical 2024) without AGI could still develop into something better.

But what about the actual future?

It clearly seems very likely that AI progress will continue. Median respondents to the 2023 Expert Survey on Progress in AI "put 5% or more on advanced AI leading to human extinction or similar, and a third to a half of participants gave 10% or more". Some people seem to think that the extinction event that is expected with some 5% or whatever in the AI catastrophe case is some very fast event - maybe too fast for people to even realize what is happening. I do not know why that should be the case; a protracted and very unpleasant catastrophe seems at least as likely (conditional on the extinction case). So these 5% do not seem negligible.[1]

Well, at least in 19 of 20 possible worlds, everything goes extremely well because we have a benevolent AGI then, right?

That's not clear, because an AGI future seems hard to imagine anyway. It seems so hard to imagine that while I've read a lot about what could go wrong, I haven't yet found a concrete scenario of a possible fut...
Causal Graphs of GPT-2-Small's Residual Stream
by David Udell — 2024-07-09T22:06:55.775Z

Thanks to the many people I've chatted with this about over the past many months. And special thanks to Cunningham et al., Marks et al., Joseph Bloom, Trenton Bricken, Adrià Garriga-Alonso, and Johnny Lin, for crucial research artefacts and/or feedback.

Codebase: sparse_circuit_discovery

TL; DR: The residual stream in GPT-2-small, expanded with sparse autoencoders and systematically ablated, looks like the working memory of a forward pass. A few high-magnitude features causally propagate themselves through the model during inference, and these features are interpretable. We can see where in the forward pass, due to which transformer layer, those propagating features are written in and/or scrubbed out.

Introduction

What is GPT-2-small thinking about during an arbitrary forward pass?

I've been trying to isolate legible model circuits using sparse autoencoders. I was inspired by the following example, from the end of Cunningham et al. (2023):

This directed graph is a legible circuit spread across a naturalistic transformer. It is a best-case interpretability success for sparse autoencoders and circuit discovery.

I wanted to see whether naturalistic transformers[1] are generally this interpretable as circuits under sparse autoencoding. If this level of interpretability just abounds, then high-quality LLM mindreading & mindcontrol is in hand! If not, could I show how far we are from that kind of mindreading technology?

Related Work

As mentioned, I was led into this project by Cunningham et al. (2023), which established key early results about sparse autoencoding for LLM interpretability.

While I was working on this, Marks et al. (2024) developed an algorithm approximating the same causal graphs in constant time. Their result is what would make this scalable and squelch down the iteration loop on interpreting forward passes.

Methodology

A sparse autoencoder is a linear map, whose shape is (autoencoder_dim, model_dim). I install sparse autoencoders at all of GPT-2-small's residual streams (one per model layer, 12 in total). Each sits at a pre_resid bottleneck that all prior information in that forward pass routes through.[2]

I fix a context, and choose one forward pass of interest in that context. In every autoencoder, I go through and independently ablate out all of the dimensions in autoencoder_dim during a "corrupted" forward pass. For every corrupted forw...
If AI starts to end the world, is suicide a good idea?
by IlluminateReality — 2024-07-09T21:53:23.470Z

For a while I’ve thought to myself that if AI starts to obviously end the world I would just commit suicide, mainly to avoid any potential s-riskiness. But I’ve become far less certain recently that that would be a good idea.
Between the possibility of resurrection, quantum immortality, and weird acausal shenanigans, I’m not sure what my plan is for if the nano-factories starting popping up and whatnot.

This uncertainty about what I should do in such a situation causes me more discomfort than having a plan, even if it’s grim.

What do you think is the right thing to do if it becomes obvious that the world is ending due to AI?
Rationalist Purity Test 
by Gunnar_Zarncke — 2024-07-09T20:30:05.421Z
Link: https://ratpuritytest.com

A quick, fun, and somewhat random test with 100 questions that you can all see on a single page, thus easy to judge if you like it. I got 50, but now clue if that's average or not.

Seen on Twitter.
That which can be destroyed by the truth, should be assumed to should be destroyed by it
by  — 2024-07-09T19:39:57.887Z

This is a post about bringing a concept from German law, Beweislast, which is most likely also present in all legal systems worldwide, to rationality, specifically to this old but central tenant of it.

Beweislast means burden of proof, and is the framework under which it is decided that who has to proove what before court. If Kurt says Kevin hit him, and neither has any evidence, this case goes to Kurt if Kevin has the burden of proof and to Kevin if Kurt has the burden of proof.

Beweislast is the obligation to prove, that reality aligns with your version of the story more, than that of the other guy. Having the Beweislast is a massive drawback, as it forces you to argument for a point, while the other guy can just sit back and try to dismantle your arguments.

While Beweislast is bad for you, it is of course not the end of all things, if Kevin has the Beweislast, but he has evidence that Kurt hit him, then Kevin will win the case. He had the burden of proof, and he delivered on it.

Now there is a really simple rule of thumb as for who should get the Beweislast, Everyone has to prove that which benefits them.

This rule of thumb alone is probably worth internalising, as it is highly effective at producing accurate legal judgments, and as such is good for human relations, or more broadly conflict mediation, in general.

In some cases however the Beweislast is inverted. German has the cute name Beweislastumkehr for this, literally inversion of burden of proof. This is for cases in which it is generally unfair for the party to prove their version of events, the most common case for in German law being business-customer contracts. If you as a customer purchase something from a business, then you are usually buying from a huge corporation, and if something breaks, it will be really hard and annoying for you to prove it was their fault. In this case there is a Beweislastumkehr, and the business has to prove that it was your fault that the thing broke, and if not, they have to send you a new one. (this is limited to six months to avoid some shenanigans).

Why am I telling you about Beweislast and Beweislastumkehr?

Because I think it solves the second virtue of rationality.

Relinquishment wants you to destroy all beliefs which are not aligned with reality with the truth, no matter the circumstance. "Rules without exceptions last eternally", sings on...
AISN #38: Supreme Court Decision Could Limit Federal Ability to Regulate AI
Plus, “Circuit Breakers” for AI systems, and updates on China’s AI industry
by Corin Katzke — 2024-07-09T19:28:29.338Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-38-supreme-court

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

Supreme Court Decision Could Limit Federal Ability to Regulate AI

In a recent decision, the Supreme Court overruled the 1984 precedent Chevron v. Natural Resources Defence Council. In this story, we discuss the decision’s implications for regulating AI.

Chevron allowed agencies to flexibly apply expertise when regulating. The “Chevron doctrine” had required courts to defer to a federal agency’s interpretation of a statute in the case that that statute was ambiguous and the agency’s interpretation was reasonable. Its elimination curtails federal agencies’ ability to regulate—including, as this article from LawAI explains, their ability to regulate AI.

The Chevron doctrine expanded federal agencies’ ability to regulate in at least two ways. First, agencies could draw on their technical expertise to interpret ambiguous statutes rather than rely on lawmakers or courts to provide clarity. Second, they could more easily apply existing statutes to emerging areas of regulation.

The loss of Chevron will be particularly impactful for AI regulation. AI is a technical and rapidly changing industry. More than most areas of regulation, then, federal agencies will require technical expertise and flexibility to effectively regulate AI. The loss of Chevron makes the efforts of AI regulators vulnerable to lengthy and uncertain challenges in the courts.

Lawmakers should create AI-specific legislation that explicitly grants agencies regulatory discretion. Federal agencies will no longer be able to easily apply existing legislation to emerging domains. Instead, the end of Chevron creates an onus on US lawmakers to develop new, AI-specific legislation to enable regulating authorities—for example, California’s SB 1047.

Lawmakers should also explicitly grant federal agencies broad discretion to interpret key parts of such legislation—for example, the definition of a “frontier model.”

“Circuit Breakers” for AI Systems

LLMs and other AI systems are vulnerable to adversarial attacks. For example, users can often get LLMs to generate harmful outputs through jailbreaking. While several approaches have been proposed to defend against these kinds of attacks, they fai...
Summer Tour Stops
by jefftk — 2024-07-09T19:10:05.659Z

Last week, Lily, Anna, Cecilia, Harris, and I drove to St
Louis and back playing dances. I wrote about one aspect

last week, and will
probably continue splitting thoughts across posts. Today I'm thinking
some about our approach to stopping along the way.

I grew up in a family with a very strong "maximize time at
destination" approach to road trips. We would often leave before
daylight, with stops determined by the minivan's requirements for
additional gasoline. We would eat while driving, I recall
creative efforts to handle certain biological needs, and we'd
generally try to get the trip over with as quickly as possible.

I did sometimes see other people taking a different approach, but it
wasn't until the first time I drove a group of college classmates a
long way that I realized people have widely divergent preferences
here. We were driving a van to Dance Flurry, about five hours North
of Swarthmore, and around lunchtime people started talking about maybe
stopping for lunch. I (really!) thought they were joking, and said
that since we still had more gas and had brought along a bunch of food
we didn't need to stop. There was a revolt, and we did not end up
eating while driving.

Over time, I've come around to a view where one of the great things
about traveling long distances by car is the opportunity for
interesting stops. It's often the case that there are intermediate
locations that are better places to spend the marginal hour than your
origin or destination.

Before this tour we put some time into thinking about nice places to
stop, and we also did some looking up options while driving. We ended
up stopping at:

Cecilia's father's violin shop, where he showed us
how he repairs instruments:

Lily and Anna both got to take turns putting on clamps:

Niagara
Falls, on the way from Rochester to Pittsburgh. I'd been
there once before, ten
years earlier with Amy and Audrey, but it
was great to take the older two kids there.

We also had a lot
more time than on our previous visit, and so were able to take
the elevator and boardwalk down to the base of the falls.

Lily loved the powerful wind and spray; Anna wanted to stay a bit
further from the most intense area.

Golden
Gate Park in Brookville OH on the way from Pittsburgh to
Bloomington. Really b

...
Fix simple mistakes in ARC-AGI, etc.
by Oleg Trott — 2024-07-09T17:46:50.364Z

ARC-AGI is a diverse artificial dataset that aims to test general intelligence. It's sort of like an IQ test that's played out on rectangular grids.

Last month, @ryan_greenblatt proposed an approach that used GPT-4o to generate about 8000 Python programs per task. It then selected the programs that worked on the "training" examples given, and ran them to actually solve the "test" query.

His approach achieved 72% accuracy on the part of the benchmark that humans have been measured to get 85% accuracy on.

I have an idea for an improvement, on top of this approach. It should be relatively cheap. I don't have time to work on this myself, but I hope someone else runs with it, hence this post.

The motivation for this idea is Ryan's note

[GPT-4o] makes simple mistakes like off-by-one errors extremely often

My idea is to try to fix them automatically. I call it Program Dithering. You go through the generated Python programs, and try to perturb all integer constants in it, one at a time, and maybe several at a time.

Thus, if you try two perturbations at a time, a program that looks like this

x = 7
...
y = x + 3

can become

x = 8
...
y = x + 2

etc., generating a potentially large number of candidate programs without any extra GPT-4o calls. One could also consider perturbing array indexing locations in a similar way.

If off-by-one errors are extremely common, Program Dithering could fix some or many of them, and improve the overall accuracy.

Off-by-one errors seem like a general flaw, so fixing them should not be "overfitting" the benchmark.

Generalizations:

If there are other simple mistakes that GPT-4o tends to make, e.g. swapping array indexes, one can extend the approach to try to fix them also.

Other tasks, like what AlphaCode does, might find this useful too.
Paper Summary: The Effects of Communicating Uncertainty on Public Trust in Facts and Numbers
by Jeffrey Heninger — 2024-07-09T16:50:05.776Z

by Anne Marthe van der Bles, Sander van der Linden, Alexandra L. J. Freeman, and David J. Spiegelhalter. (2020)

https://www.pnas.org/doi/pdf/10.1073/pnas.1913678117.

Summary: Numerically expressing uncertainty when talking to the public is fine. It causes people to be less confident in the number itself (as it should), but does not cause people to lose trust in the source of that number.

Uncertainty is inherent to our knowledge about the state of the world yet often not communicated alongside scientific facts and numbers. In the “posttruth” era where facts are increasingly contested, a common assumption is that communicating uncertainty will reduce public trust. However, a lack of systematic research makes it difficult to evaluate such claims.

Within many specialized communities, there are norms which encourage people to state numerical uncertainty when reporting a number. This is not often done when speaking to the public. The public might not understand what the uncertainty means, or they might treat it as an admission of failure. Journalistic norms typically do not communicate the uncertainty.

But are these concerns actually justified? This can be checked empirically. Just because a potential bias is conceivable does not imply that it is a significant problem for many people. This paper does the work of actually checking if these concerns are valid.

Van der Bles et al. ran five surveys in the UK with a total n = 5,780. A brief description of their methods can be found in the appendix below.

Respondents’ trust in the numbers varied with political ideology, but how they reacted to the uncertainty did not.

People were told the number either without mentioning uncertainty (as a control), with a numerical range, or with a verbal statement that uncertainty exists for these numbers. The study did not investigate stating p-values for beliefs. Exact statements used in the survey can be seen in Table 1, in the appendix.

The best summary of their data is in their Figure 5, which presents results from surveys 1-4. The fifth survey had smaller effect sizes, so none of the shifts in trust were significant.

Expressing uncertainty made it more likely that people perceived uncertainty in the number (A). This is good. When the numbers are uncertain, science communicators should want people to believe that they are uncertain. Interestingly, verbally reminding people of uncertai...
UC Berkeley course on LLMs and ML Safety
by Dan H — 2024-07-09T15:40:00.920Z
Link: https://rdi.berkeley.edu/understanding_llms/s24

The UC Berkeley course I co-taught now has lecture videos available: https://www.youtube.com/playlist?list=PLJ66BAXN6D8H_gRQJGjmbnS5qCWoxJNfe

Course site: Understanding LLMs: Foundations and Safety

Unrelatedly, a more conceptual AI safety course has its content available at https://www.aisafetybook.com/
What and Why: Developmental Interpretability of Reinforcement Learning
by Garrett Baker — 2024-07-09T14:09:40.649Z

Introduction

I happen to be in that happy stage in the research cycle where I ask for money so I can continue to work on things I think are important. Part of that means justifying what I want to work on to the satisfaction of the people who provide that money.

This presents a good opportunity to say what I plan to work on in a more layman-friendly way, for the benefit of LessWrong, potential collaborators, interested researchers, and funders who want to read the fun version of my project proposal

It also provides the opportunity for people who are very pessimistic about the chances I end up doing anything useful by pursuing this to have their say. So if you read this (or skim it), and have critiques (or just recommendations), I'd love to hear them! Publicly or privately.

So without further ado, in this post I will be discussing & justifying three aspects of what I'm working on, and my reasons for believing there are gaps in the literature in the intersection of these subjects that are relevant for AI alignment. These are:

Reinforcement learning
Developmental Interpretability
Values

Culminating in: Developmental interpretability of values in reinforcement learning.

Here are brief summaries of each of the sections:

Why study reinforcement learning?

Imposed-from-without or in-context reinforcement learning seems a likely path toward agentic AIs
The “data wall” means active-learning or self-training will get more important over time
There are fewer ways for the usual AI risk arguments to fail in the RL with mostly outcome-based rewards circumstance than the supervised learning + RL with mostly process-based rewards (RLHF) circumstance.

Why study developmental interpretability?

Causal understanding of the training process allows us to produce reward structure or environmental distribution interventions
Alternative & complementary tools to mechanistic interpretability
Connections with singular learning theory

Why study values?

The ultimate question of alignment is how can we make AI values compatible with human values, yet this is relatively understudied.

Where are the gaps?

Many experiments
Many theories
Few experiments testing theories or theories explaining experiments

Reinforcement learning

Agentic AIs vs Tool AIs

All generally capable adaptive systems are ruled by a general, ground-truth, but slow outer optimization process which reduces incoherency and co...
Medical Roundup #3
by Zvi — 2024-07-09T13:10:06.862Z

This time around, we cover the Hanson/Alexander debates on the value of medicine, and otherwise we mostly have good news.

Technology Advances

Regeneron administers a single shot in a genetically deaf child’s ear, and they can hear after a few months, n=2 so far.

Great news: An mRNA vaccine in early human clinical trials reprograms the immune system to attack glioblastoma, the most aggressive and lethal brain tumor. It will now proceed to Phase I. In a saner world, people would be able to try this now.

More great news, we have a cancer vaccine trial in the UK.

And we’re testing personalized mRNA BioNTech canner vaccines too.

US paying Moderna $176 million to develop a pandemic vaccine against bird flu.

We also have this claim that Lorlatinib jumps cancer PFS rates from 8% to 60%.

The GLP-1 Revolution

Early results from a study show the GLP-1 drug liraglutide could reduce cravings in people with opioid use disorder by 40% compared with a placebo. This seems like a clear case where no reasonable person would wait for more than we already have? If there was someone I cared about who had an opioid problem I would do what it took to get them on a GLP-1 drug.

Rumblings that GLP-1 drugs might improve fertility?

Rumblings that GLP-1 drugs could reduce heart attack, stroke and death even if you don’t lose weight, according to a new analysis? Survey says 6% of Americans might already be on them. Weight loss in studies continues for more than a year in a majority of patients, sustained up to four years, which is what they studied so far.

The case that GLP-1s can be sued against all addictions at scale. It gives users a sense of control which reduces addictive behaviors across the board, including acting as a ‘vaccine’ against developing new addictions. It can be additive to existing treatments. More alcoholics (as an example) already take GLP-1s than existing indicated anti-addiction medications, and a study showed 50%-56% reduction in risk of new or recurring alcohol addictions, another showed 30%-50% reduction for cannabis.

How to cover this? Sigh. I do appreciate the especially clean example below.

Matthew Yglesias: Conservatives more than liberals will see the systematic negativity bias at work in coverage of GLP-agonists.

Less likely to admit that this same dynamic colors everything including coverage of crime and the economy.

The situation is tha...