trentmkelly/LessWrong-43k huggingface dataset:
Announcing SPAR Summer 2024!
by laurenmarie12 — 2024-04-16T08:30:31.339Z

The

Supervised Program for Alignment Research

(SPAR) is open to applications for summer 2024! SPAR connects mentors and mentees for three-month AI safety and governance research projects. The

Summer 2024 cycle of SPAR

will take place virtually between June 14th and September 10th, 2024.

Fill out this

interest form

if you would like to be updated about important SPAR deadlines for current and future SPAR iterations!

SPAR mentors

propose and supervise projects while managing a team of mentees. You might be a good fit to be a SPAR mentor if you are a graduate student, academic, full-time AI safety researcher, independent researcher, or have prior full-time relevant research experience (e.g., MATS, Astra, GovAI fellow, etc.).

Apply here

by April 26th!

SPAR mentees

participate in a collaborative research project under an experienced mentor. You might be a good fit to be a SPAR mentee if you have a technical/STEM background or a governance/policy background. SPAR projects vary in how much alignment-specific knowledge they require. Mentee applications open on April 29th.

Apply here

by May 24th.

SPAR was originally organized by Stanford AI Safety, Berkeley AI Safety Initiative, and Georgia Tech AI Safety, but is now expanding its scope beyond university students. The program is now run by a small team outside the university organizing ecosystem. We are actively looking for ways to make SPAR a better program, so if you have input, please let us know

here

!
We think SPAR's value add is that it enables mentors to increase their research output by working with a team of mentees, while also allowing them to develop project management, leadership, and mentorship skills. Simultaneously, mentees can explore their fit for alignment research on a flexible basis and build alignment-relevant skills. Both mentors and mentees get an opportunity to produce concrete work that serves as a signal for future hiring.

Applications for mentors are due by April 26th, and mentees are due by May 24th. You can apply to be a SPAR mentor

here

and a SPAR mentee

here

. If you are unsure about your fit, please err on the side of applying. We are also especially excited to see applications from women and other underrepresented groups! You can refer others who you think might be a good fit through

th

...
The argument for near-term human disempowerment through AI
by Chris_Leong — 2024-04-16T04:50:53.828Z
Link: https://link.springer.com/article/10.1007/s00146-024-01930-2

Author: Leonard Dung
Abstract: Many researchers and intellectuals warn about extreme risks from artificial intelligence. However, these warnings typically came without systematic arguments in support. This paper provides an argument that AI will lead to the permanent disempowerment of humanity, e.g. human extinction, by 2100. It rests on four substantive premises which it motivates and defends: first, the speed of advances in AI capability, as well as the capability level current systems have already reached, suggest that it is practically possible to build AI systems capable of disempowering humanity by 2100. Second, due to incentives and coordination problems, if it is possible to build such AI, it will be built. Third, since it appears to be a hard technical problem to build AI which is aligned with the goals of its designers, and many actors might build powerful AI, misaligned powerful AI will be built. Fourth, because disempowering humanity is useful for a large range of misaligned goals, such AI will try to disempower humanity. If AI is capable of disempowering humanity and tries to disempower humanity by 2100, then humanity will be disempowered by 2100. This conclusion has immense moral and prudential significance.

My thoughts: I read through it rather quickly so take what I say with a grain of salt. That said, it seemed persuasive and well-written. Additionally, the way that they split up the argument was quite nice.
My experience using financial commitments to overcome akrasia
by William Howard — 2024-04-15T22:57:32.574Z

About a year ago I decided to try using one of those apps where you tie your goals to some kind of financial penalty. The specific one I tried is Forfeit, which I liked the look of because it’s relatively simple, you set single tasks which you have to verify you have completed with a photo.

I’m generally pretty sceptical of productivity systems, tools for thought, mindset shifts, life hacks and so on. But this one I have found to be really shockingly effective, it has been about the biggest positive change to my life that I can remember. I feel like the category of things which benefit from careful planning and execution over time has completely opened up to me, whereas previously things like this would be largely down to the luck of being in the right mood for long enough.

It’s too soon to tell whether the effect will fade out eventually, but I have been doing this for ~10 months now[1] so I think I’m past the stage of being excited by a new system and can in good conscience recommend this kind of commitment mechanism as a way of overcoming akrasia.

The rest of this post consists of some thoughts on what I think makes a good akrasia-overcoming approach in general, having now found one that works (see hindsight bias), and then advice on how to use this specific app effectively. This is aimed as a ~personal reflections post~ rather than a fact post.

Thoughts on what makes a good anti-akrasia approach

I don’t want to lean too much on first principles arguments for what should work and what shouldn’t, because I was myself surprised by how well setting medium sized financial penalties worked for me. I think it’s worth explaining some of my thinking though, because the advice in the next section probably won’t work as well for you if you think very differently.

1. Behaviour change (“habit formation”) depends on punishment and reward, in addition to repetition

A lot of advice about forming habits focuses on the repetition aspect, I think positive and negative feedback is much more important.

One way to see this is to think of all the various admin things that you put off or have to really remind yourself to do, like taking the bins out. Probably you have done these hundreds or thousands of times in your life, many more times than any advice would recommend for forming a habit. But they are boring or unpleasant every time so you have to layer ...
An evaluation of circuit evaluation metrics
by Iván Arcuschin — 2024-04-15T19:38:53.457Z

This work was produced as part of the ML Alignment & Theory Scholars Program - Winter 2023-24 Cohort, with mentorship from Adrià Garriga-Alonso. Equal contribution by Niels and Iván.

Summary

In this post we analyse the circuit evaluation metrics proposed by

Wang et al.

(IOI paper) and show that their specific formalisation allows for counter intuitive examples that don't always reflect the natural language properties they aim to capture. For example, some incomplete circuits can be made complete by removing components, and there exist faithful, complete and minimal circuits that have smaller circuits that are also faithful, complete and minimal.

Introduction

The main goal of Mechanistic Interpretability is to reverse engineer neural networks to make the learned weights into human-interpretable algorithms, hoping to develop a more principled understanding of deep learning. In this setup, models are seen as computational graphs, and circuits are subgraphs with distinct functionality. Circuits serve as a critical unit of measure to understand neural networks, since they allow us to analyse a model's behaviour only on a subset of its capabilities, usually by constraining the space of inputs to the ones associated with a specific task (e.g., the Indirect Object Identification task, as described by

Wang et al.

). Although circuits can be labelled, by assigning a human interpretation to some of its parts, in this post we focus on unlabelled ones.

In recent years, several techniques have been proposed for finding circuits (either manually or automatically). That is, given a specific task for which the model has a good performance, find the circuit (subgraph in the model) that is responsible for it.

Evaluation metrics

As part of the circuit discovery techniques, researchers have also designed and proposed circuit evaluation metrics that measure whether a circuit is "good" or not. However, although the natural language interpretation of these metrics usually seems reasonable, their specific formalisation can sometimes lead to counter intuitive results as we will show. In the following sections, we describe and present limitations for the three metrics proposed by

Wang et al.

: faithfulness, completeness, and minimality. Further details and specific examples can be found in the Appendix.

Notation: M is the full model,&nb...
Experiments with an alternative method to promote sparsity in sparse autoencoders
by Eoin Farrell — 2024-04-15T18:21:48.771Z

Summary

I experimented with alternatives to the standard L1 penalty used to promote sparsity in sparse autoencoders (SAEs). I found that including terms based on an alternative differentiable approximation of the feature sparsity in the loss function was an effective way to generate sparsity in SAEs trained on the residual stream of GPT2-small. The key findings include:

SAEs trained with this new loss function had a lower L0 norm, lower mean-squared error, and fewer dead features compared to a reference SAE trained with an L1 penalty.
This approach can effectively discourage the production of dead features by adding a penalty term to the loss function based on features with a sparsity below some threshold.
SAEs trained with this loss function had different feature sparsity distributions and significantly higher L1 norms compared to L1-penalised models.

Loss functions that incorporate differentiable approximations of sparsity as an alternative to the standard L1 penalty appear to be an interesting direction for further investigation.

Motivation

Sparse autoencoders (SAEs) have been shown to be effective at extracting interpretable features from the internal activations of language models (e.g.

Anthropic

&

Cunningham et al.

). Ideally, we want SAEs to simultaneously (a) reproduce the original language model behaviour and (b) to consist of monosemantic, interpretable features. SAE loss functions usually contain two components:

Mean-squared error (MSE) between the SAE output and input activations, which helps with reconstructing the original language model activations, and ultimately with model behaviour.
L1 penalty on the SAE feature activations (the sum of the magnitude of the feature activations) to promote sparsity in the learned representation.

The relative importance of each term is controlled by a coefficient on the L1 penalty, which allows the model to move along the trade-off between reconstruction of the language model behaviour and a highly sparse representation. In this post, I present experiments with alternatives to the standard L1 penalty to promote sparsity in SAEs.

Approximations of the sparsity

A key requirement for SAE features to be interpretable is that most of them are sparse. In this context, the sparsity, s, of a given SAE feature, f, is the fraction of tokens for which the feature has a nonzero activation. For ...
Effectively Handling Disagreements - Introducing a New Workshop
by Camille Berger  — 2024-04-15T16:33:50.339Z

On May 25th, 2023, someone posted a review of How Minds Change on LessWrong. It talked about Street Epistemology, Deep Canvassing, and Smart Politics, ways of handling disagreements that open the possibility of rational belief progression through amicable discussions. Summarized quickly, they rely on active listening, sharing personal stories and socratic questioning.

You can now learn all of those three techniques online, for free, in 4 hours, and in a Deliberate Practice setting. If interested, you can also learn them in an in-person workshop spanning anytime between 2 hours and a full weekend -just shoot me an email with the object EHD (at the time of writing, I’m based in Paris, France).

You can enroll on the website (see bottom for subscribing to the mailing list), and join the discord server.

About the workshop:

What would you learn?

When you find yourself in disagreement with someone on a significant issue, and they might not share your perspectives or even show resistance towards them, it's natural to seek a productive dialogue. The goal is to have a conversation that brings both parties closer to understanding the truth. However, jumping directly into counter-arguments often proves counterproductive, leading to further resistance or increasingly complex counterpoints. It's easy to label the other person as "irrational" in these moments.

To navigate these conversations more effectively, I'm offering a workshop that introduces a range of techniques based on evidence and mutual agreement. These methods are designed to facilitate discussions about deeply held beliefs in a friendly manner, keeping the focus on the pursuit of truth.
Techniques are the following:

4h version:

Deep Canvassing
Street Epistemology
Narrative Transportation
Cooling Conversations (Smart Politics)

12h version:
All the aforementioned plus Principled Negotiation and bits of Motivational Interviewing

Who is this for?

I’m mainly targeting people who are not used to such interactions, or feel frustrated by them -as such, you might not learn a lot if you are already used to managing high-stakes interactions.

In the specific case of Rationality/EA, this would allow you to :

Expand the community’s awareness by easing exchanges with outsiders
e.g. if you are a professional researcher in AI Safety wanting to discuss with other researchers who are skeptical of your field.
Carefully spread awareness

...
Four Local Gigs
by jefftk — 2024-04-15T16:00:02.389Z

I haven't been playing in Boston much lately, but over the next month
I'll be playing four times:

2024-04-19 through 2024-04-21: NEFFA. It's this
weekend! On Friday night Kingfisher is playing the
contra
medley (9pm) and then a set
of contras (9:30) with Alex
Deis-Lauby calling. We'll be playing again on Sunday for a family
dance (3pm) with Kristin
Siebert calling. Plus Lily and I are leading a kids
contra tune jam on Sunday at 2pm (tunes).

2024-04-27: Northboro. Adults $10, $5
students. Rick, Lily, and I are playing as Dandelion. This is Lily's
first dance as a booked musician, and first time playing on-mic! It's
also an afternoon dance, 2pm-4:30, not an evening one. Harris Lapiroff will be
calling.

2024-05-11: Somerville
Porchfest on Morrison Ave, Kingfisher with Harris calling. Free
and outdoors, with a rain date of the next day. Here's the status of
my block party application, with all the different city departments
reviewing it; it looks likely that we'll be able to close the street
again:

We're playing 2pm until ~3:30pm, since we need to drive to Hartford to play the
evening dance with Don Veino.

2024-05-16: Concord Thursday.
Evening contra dance as Kingfisher with Julian Blechner
calling.

Comment via: facebook, lesswrong, mastodon
Taking into account preferences of past selves
by Jacob G-W — 2024-04-15T13:15:10.545Z

Problem

“You’ll have a great time wherever you go to college!” I constantly hear this. From my parents, my friends’ parents, my guidance counselor, and my teachers. I don’t doubt it. I’m sure I’ll have a lot of fun wherever I go. Since I’m trying to be very intentional about my college decision process, I’ve interviewed close to twenty students. And for the most part, all of them are having a great time!

This scares me. A lot.

If I can go anywhere and have a great time, then what should I choose my college based on? Ranking? Prestige? Food? Campus? Job opportunities? Cost?

After thinking more about this problem, I realized that although I’ll have fun wherever I’ll go, it will also change me as a person. More specifically, I think that each college will shift my values in a different way.

Value shift is when, through one event or another, a person changes what they value. Their past self, seeing their future self with different values, might think “oh no, I have become what I sought to destroy.” Milder versions also exist. The classic example is what happens to a parent after they have a child. If they had a value of “stay alive” before, it (usually) changes to “keep child alive then stay alive.” Yet if you had told them in the past that they would sacrifice themself for their child, they might not be okay with it (or they might be, in which case a value shift did not happen). The next question would be: why did you take an action (having children) that would shift your terminal values?

So when I look out over the imaginary cause and effect tree of me choosing colleges, I see further. Not only do I see the effects of

how I will spend the next four years?
what people I will meet that I will know for the rest of my life?
what skills will I learn for the rest of my life?
what meta-skills (or mannerisms/habits of being) will I pick up?

I also see the effect of

5. what will I be motivated to do with my life?

I worry that the last one is the most important one, yet also the most overlooked. Naively, I could say that my future self will obviously be wiser than my present self in choosing what to do with my life so I should not worry at all. After all, it has all of my experience plus four extra years! Jacob in four years literally was me who was thinking these thoughts. Of course he won’t do something stupid.

And if I were perfectly rational agent, I’d say tha...
Monthly Roundup #17: April 2024
by Zvi — 2024-04-15T12:10:03.126Z

As always, a lot to get to. This is everything that wasn’t in any of the other categories.

Bad News

You might have to find a way to actually enjoy the work.

Greg Brockman (President of OpenAI): Sustained great work often demands enjoying the process for its own sake rather than only feeling joy in the end result. Time is mostly spent between results, and hard to keep pushing yourself to get to the next level if you’re not having fun while doing so.

Yeah. This matches my experience in all senses. If you don’t find a way to enjoy the work, your work is not going to be great.

This is the time. This is the place.

Guiness Pig: In a discussion at work today:

“If you email someone to ask for something and they send you an email trail showing you that they’ve already sent it multiple times, that’s a form of shaming, don’t do that.”

Others nodding in agreement while I try and keep my mouth shut.

JFC…

Goddess of Inflammable Things: I had someone go over my head to complain that I was taking too long to do something. I showed my boss the email where they had sent me the info I needed THAT morning along with the repeated requests for over a month. I got accused by the accuser of “throwing them under the bus”.

You know what these people need more of in their lives?

Jon Stewart was told by Apple, back when he had a show on AppleTV+, that he was not allowed to interview FTC Chair Lina Khan.

This is a Twitter argument over whether a recent lawsuit is claiming Juul intentionally evaded age restrictions to buy millions in advertising on websites like Nickelodeon and Cartoon Network and ‘games2girls.com’ that are designed for young children, or whether they bought those ads as the result of ‘programmatic media buyers’ like AdSense ‘at market price,’ which would… somehow make this acceptable? What? The full legal complaint is here. I find it implausible that this activity was accidental, and Claude agreed when given the text of the lawsuit.

I strongly agree with Andrew Sullivan, in most situations playing music in public that others can hear is really bad and we should fine people who do it until they stop. They make very good headphones, if you want to listen to music then buy them. I am willing to make exceptions for groups of people listening together, but on your own? Seriously, what the hell.

Democrats somewhat souring on all of electric cars, ...
Reconsider the anti-cavity bacteria if you are Asian
by Lao Mein — 2024-04-15T07:02:02.655Z

Update: further research has led me to believe that people of all races should test themselves for ALDH deficiency before using Lumina. Even if you don't exhibit AFR symptoms when drinking alcohol, your ALDH activity may still be decreased.

Many people in the rational sphere have been promoting Lumina/BCS3-L1, a genetically engineered bacterium, as an anti-cavity treatment. However, none have brought up a major negative interaction that may occur with a common genetic mutation.

In short, the treatment works by replacing lactic acid generating bacteria in the mouth with ones that instead convert sugars to ethanol, among other changes. Scott Alexander made a pretty good FAQ about this. Lactic acid results in cavities and teeth demineralization, while ethanol does not. I think this is a really cool idea, and would definitely try it if I didn't think it would significantly increase my chances of getting oral cancer.

Why would that be? Well, I, like around half of East Asians, have a mutation in my acetaldehyde dehydrogenase (ALDH) which results in it being considerably less active. This is known as Asian/Alcohol Flush Reaction (AFR). This results in decreased ability to metabolize acetaldehyde to acetate and consequently a much higher level of acetaldehyde when drinking alcohol. Although the time ingested ethanol spends in the mouth and stomach are quite short, alcohol dehydrogenase activity by both human and bacterial cells rises rapidly once the presence of ethanol is detected. Some studies have estimated that ~20% of consumed ethanol is converted to acetaldehyde in the mouth and stomach in a process called first pass metabolism. Normally, this is broken down into acetate by the ALDH also present, but it instead builds up in those with AFR. Acetaldehyde is a serious carcinogen and people with AFR have significantly higher levels of oral and stomach cancer (The odds ratios for Japanese alcoholics with the mutation in relation to various cancers are >10 (!!!) for oral and esophageal cancer). The Japanese paper also notes that all alcoholics tested only had a single copy of the mutation, since it is very difficult to become an alcoholic with two copies (imagine being on high dosage Antabuse your entire life - that's the same physiological effect).

In addition, there is also the potential for change in oral flora and their resting ADH levels. As oral flora an...