trentmkelly/LessWrong-43k huggingface dataset:
SAE Probing: What is it good for? 
by Subhash Kantamneni — 2024-11-01T19:23:55.418Z

Subhash and Josh are co-first authors. Work done as part of the two week research sprint in Neel Nanda’s MATS stream

Update February 2025:

We have recently expanded this post into a full paper: https://arxiv.org/abs/2502.16681

Our results are now substantially more negative. We find that SAE probes do not consistently beat baselines on test AUC and are not a useful addition to a practitioner's toolkit. We also found that baseline logistic regression probes worked as well even on the interpretability case studies that we were initially most excited about. We also tried multi-token probing and found that although SAE probes improved with max or mean aggregation, an attention-based logistic regression probe removed most of the advantage.

*The old title of this post was: SAE Probing: What is it good for? Absolutely something!

TLDR

We show that dense probes trained on SAE encodings are competitive with traditional activation probing over 60 diverse binary classification datasets
Specifically, we find that SAE probes have advantages with:

Low data regimes (~ < 100 training examples)
Corrupted data (i.e. our training set has some incorrect labels, while our test set is clean)
Settings where we worry about the generalization of our probes due to spurious correlations in our dataset or possible mislabels (dataset interpretability) or if we want to understand SAE features better (SAE interpretability).

We find null results when comparing SAE probes to activation probes with OOD data in other settings or with imbalanced classes.
We find that higher width and L0 are determining factors for which SAEs are best for SAE probing.

Motivation and Problem Setting

Sparse autoencoders (SAEs) have quickly become the hot topic in mechanistic interpretability research, but do they have any actual use cases? We argue probing might be one! The intuition is that if SAEs are doing what we want them to and finding meaningful model variables, we should be able to use them as a strong inductive bias for training good linear probes. Anthropic recently released an update exploring a similar idea, but only explored variations on one dataset in a single setting. We collected over 60 binary classification datasets (described here). These datasets include sentiment classification, linguistic acceptability, differentiating ChatGPT and human written text, identifying...
Set Theory Multiverse vs Mathematical Truth - Philosophical Discussion
by Wenitte Apiou — 2024-11-01T18:56:06.900Z

I've been thinking about the set theory multiverse and its philosophical implications, particularly regarding mathematical truth. While I understand the pragmatic benefits of the multiverse view, I'm struggling with its philosophical implications.

The multiverse view suggests that statements like the Continuum Hypothesis aren't absolutely true or false, but rather true in some set-theoretic universes and false in others. We have:

Gödel's Constructible Universe (L) where CH is true

Forcing extensions where CH is false

Various universes with different large cardinal axioms

However, I find myself appealing to basic logical principles like the law of non-contradiction. Even if we can't currently prove certain axioms, doesn't this just reflect our epistemological limitations rather than implying all axioms are equally "true"?

To make an analogy: physical theories being underdetermined by evidence doesn't mean reality itself is underdetermined. Similarly, our inability to prove CH doesn't necessarily mean it lacks a definite truth value.

Questions I'm wrestling with:

What makes certain axioms "true" beyond mere consistency?

Is there a meaningful distinction between mathematical existence and consistency?

Can we maintain mathematical realism while acknowledging the practical utility of the multiverse approach?

How do we reconcile Platonism with independence results?

I'm leaning towards a view that maintains objective mathematical truth while explaining why we need to work with multiple models pragmatically. But I'm very interested in hearing other perspectives, especially from those who work in set theory or mathematical logic.
Educational CAI: Aligning a Language Model with Pedagogical Theories
by Bharath Puranam — 2024-11-01T18:55:26.993Z

Bharath Puranam (bharath225525@gmail.com)

This research blog represents my final project for the AI Safety Fundamentals: AI Alignment Course for June 2024 Cohort.

Update: Finally received my certification for the course: AI Safety Fundamentals: Alignment | Bharath Puranam

Imagine a world where artificial intelligence doesn't just assist in education, but actively shapes how teenagers learn. That's the future we're exploring in this project, where we're combining time-tested pedagogical theories with cutting-edge AI.

In today's rapidly evolving digital landscape, teenagers are at the forefront of technological change. They're growing up with AI as a constant companion, from smartphone assistants to personalized learning apps. But how can we ensure that AI truly enhances their education rather than just adding to the noise?

Our project dives into this challenge head-on. We're developing a set of principles that could guide AI systems to be not just smart, but truly effective and ethical teachers for our teens. By blending established educational theories with the latest in AI ethics, we're aiming to create a framework that could revolutionize how teenagers learn in the AI age.

This blog post will take you through our journey, from the pedagogical theories we've explored to the AI principles we've developed. We'll discuss why these matters, how we did it, and what it could mean for the future of education.

Project Question

How can we use pedagogical theories to develop Constitutional AI principles to create ethical and effective AI learning systems for teenage education?

The project focuses on applying Constitutional AI principles to the context of adolescent/teenage learning, defined by the World Health Organization as individuals aged 10-19 which covers most secondary and post-secondary/tertiary level students.

Aim of the Project

The project's aim is guided by the following key questions:

Are AI systems aligned with educational goals and objectives?
Can AI systems explain their responses using predefined criteria or principles?
Do AI systems help students achieve educational goals and outcomes?
Are AI systems effectively supporting the learning process?

Pedagogical Theories TLDR

Pedagogical theories are ideas about how people learn and how teachers can help them better in their pursuit of learning. They try to explain the learning process and provide guidance on effective tea...
Prediction markets and Taxes
by Edmund Nelson — 2024-11-01T17:39:35.191Z

(Epistemic status: This is super trivial information)

(Note 2: there is a timely Culture war issue that triggered this post but I will not be mentioning any Culture war issues in this post)

Imagine you worked for the US mint, and somebody was betting that a coin you manufactured was unfair, now you work for the US mint and you know this coin is fair, it isn't weighted and the edge alignment is perfect. The person making the bet going to use a coin flipping robot to remove any human deception.

What odds would you have to be given to actually take the bet?

I'll be using American odds. So in theory you'd accept any odds above +100 right? Well in a world without taxes yes, but taxes are one sided, you don't pay negative taxes on gambling losses, but you do pay positive taxes on gambling winnings. As such we need odds that are better than +100/(100-T) where T is our marginal tax rate. Marginal tax rate means territory/state, local and Federal income taxes which gets complicated, however for now if we assume 33.3% as our marginal tax rate (California man making >100k/year) , we would need odds of +150 or better to be worth betting. (That's 60% or better in manifold terms)

I feel like this point is understated, for real money markets you need to do the following to get the "estimated odds"
Let market "probability" be = M (for these purposes the bid ask spread in most markets is trivial relative to the weight of taxes)

The actual probability that the event happens is at least

2M3−M

The probability it doesn't is at most

3MM+2

all told those are pretty stark differences, if the market is saying that an event has a 63% chance of happening then the event has anywhere from a 53-72% chance of happening!

So if somebody says "the market is giving probability of 63% but my model says it has a 55% chance of happening, "why aren't you betting" is not a credible argument because If they were right they would still be unable to profit from their disagreement with the market.
Dentistry, Oral Surgeons, and the Inefficiency of Small Markets
by GeneSmith — 2024-11-01T17:26:06.466Z

There are million dollar bills lying on the ground for those with eyes to see them.

I'm in Miami today, visiting a friend who is opening a dental practice. Chatting with someone very knowledgeable about a niche industry is an interesting experience because you get to learn about just how insane everything is behind the scenes. For example, dental practices benefit significantly from scale, yet the overwhelming majority of operations are small private practices.

My friend (let's call him Jeffrey for the purposes of this post), is likely going to be able to do a few million in revenue the first year his dentistry office is open. The facility is sized to reach $15 million in annual revenue, which they expect to be able to achieve in the next couple of years. The expected EBITA of this business (earnings before interest, taxes, and amortization) are 40-50%.

That means if all goes well (and it appears to be so far), my friend will be making 3-6 million per year in profits in a couple of years, with the possibility of much more if he's able to open additional clinics.

What's perhaps most surprising about this business is he put in very little of his own money to get it off the ground. He didn't raise very much money: $400,000, mostly from friends and family.

The rest of the required capital came from a $1.7 million bank loan. The interest rate was 6.5%.

To anyone familiar with the startup world, this sounds absurd. Banks face asymmetric risk when lending money to new companies which heavily discourages unsecured lending.

New businesses frequently fail, putting loans into default. And when they succeed the bank sees none of the upside because a loan doesn't give them ownership rights over the business.

For this reason, when banks DO lend to startups it's often in the form of a convertible loan. Convertible loans are like regular loans, except when the bill comes due the value of the loan can convert into equity in the startup.

Who exactly decides whether the loan is repaid in cash or equity can vary somewhat from one loan to the next, but in most cases convertible loans provide significant upside for the bank in situations where the new company does well.

So when my friend learned that the bank would give him an unsecured $1.7 million loan at a 6.5% interest rate for his new business, he was somewhat surprised.

Why are loans for dentistry offices so cheap?

As it turns out, den...
Live Machinery: An Interface Design Philosophy for Wholesome AI Futures
by Sahil — 2024-11-01T17:24:09.957Z

Fluid interfaces for sensemaking at pace with AI development.

Register interest here for the workshop in November 2024 (or to be invited to future workshops). To know some practical details, go to section 2.
Click here to help raise funds for the workshop and related events.
Apply for the AI Safety camp 2025 project on live machinery in tech and governance.
To learn more about the broader vision and engage with this strange community, send me a DM.

Disclaimer: this is both an invitation and something of a reference doc. I've put the more reference-y material in collapsible sections at the end. The bold font is for easier skimming.

0. What is this?

This is an extended rendezvous for creating culture and technology, for those enlivened by reimagining AI-based interfaces.

Specifically, we’re creating for the near future where our infrastructure is embedded with realistic levels of intelligence (i.e., only mildly creative but widely adopted) yet full of novel, wild design paradigms anyway.

In particular, we’re going to concretely explore wholesome interface designs that anticipate a teleattention era. Where cheap attentivity implies unstructured meanings can scalably compose without requiring shared preformed protocols, because moderate AI reliably does tailored, just-in-time “autostructuring".

The focus (which we will get to, with some patience) is on applying this to transforming research methodology and interaction. This is what “live theory” is about: sensemaking that can feed into a rich, sane, and heart-opening future.

Huh?

It’s a project for AI interfaces that don’t suck, to redo how (conceptual AI safety) research is done.

Wait, so you think AI can only be mildly intelligent?

Nope.

But you only care about the short term, of “mild intelligence”?

Nope, the opposite. We expect AI to be very, very, very transformative. And therefore, we expect intervening periods to be very, very transformative (as should you!). Additionally, we expect even “very2 transformative” intervening periods to be crucial, and quite weird themselves, however short they might be.

In preparing for this upcoming intervening period, we want to work on the newly enabled design ontologies of sensemaking that can keep pace with a world replete with AIs and their prolific outputs. Using the near-term crazy future to ...
Seeking Collaborators
by abramdemski — 2024-11-01T17:13:36.162Z

I've been accepted as a mentor for the next AI Safety Camp. You can apply to work with me on the tiling problem. The goal will be to develop reflectively consistent UDT-inspired decision theories, and try to prove tiling theorems for them.

The deadline for applicants is November 17.

The program will run from January 11 to April 27. It asks for a 10 hour/week commitment.

I am not being funded for this.[1] You can support my work on Patreon.

My project description follows:

Summary

The Tiling Agents problem (aka reflective consistency) consists of analysing when one agent (the "predecessor") will choose to deliberately modify another agent (the "successor"). Usually, the predecessor and successor are imagined as the same agent across time, so we are studying self-modification. A set of properties "tiles" if those properties, when present in both predecessor and successor, guarantee that any self-modifications will avoid changing those properties.

You can think of this as the question of when agents will preserve certain desirable properties (such as safety-relevant properties) when given the opportunity to self-modify. Another way to think about it is the slightly broader question: when can one intelligence trust another? The bottleneck for avoiding harmful self-modifications is self-trust; so getting tiling results is mainly a matter of finding conditions for trust.

The search for tiling results has four main motivations:

AI-AI tiling, for the purpose of finding conditions under which AI systems will want to preserve safety-relevant properties.
Human-AI tiling, for the purpose of understanding when we can justifiably trust AI systems.
AI-Human tiling, as a model of corrigibility. (Will AIs choose to interfere with human decisions?)
Tiling as a consistency constraint on decision theories, for the purpose of studying rationality.

These three application areas have a large overlap, and all four seem important.

The non-summary

Motivation

In the big picture, tiling seems like perhaps the single most well-motivated approach to theoretical AI safety: it allows us to directly formally address the question of how humans can justifiably trust AIs. However, extremely few people are directly working on this approach. Indeed, out of all the people who have worked on tiling in the past, the only person I’m currently aware of who continues to work on this is myself.

I think part of this...
Complete Feedback
by abramdemski — 2024-11-01T16:58:50.183Z

A simple, weak notion of corrigibility is having a "complete" feedback interface. In logical induction terms, I mean the AI trainer can insert any trader into the market. I want to contrast this with "partial" feedback, in which only some propositions get feedback and others ("latent" propositions) form the structured hypotheses which help predict the observable propositions -- for example, RL, where only rewards and sense-data is observed.

(Note: one might think that the ability to inject traders into LI is still "incomplete" because traders can give feedback on the propositions themselves, not on other traders; so the trader weights constitute "latents" being estimated. However, a trader can effectively vote against another trader by computing all that trader's trades and counterbalancing them. Of course, we can also more directly facilitate this, EG giving the user the ability to directly modify trader weights, and even giving traders an enhanced ability to bet on each other's weights.)

Why is this close to corrigibility?

The idea is that the trainer can enact "any" modification they'd like to make to the system as a trader. In some sense (which I need to articulate better), the system doesn't have any incentive to avoid this feedback.

For example, if the AI predicts that the user will soon give it the feedback that staying still and doing nothing is best, then it will immediately start staying still and doing nothing. If this is undesirable, the user can instead plan to give the feedback that the AI should "start staying still from now forward until I tell you otherwise" or some such.

This is not to say that the AI universally tries to update in whatever direction it anticipates the users might update it towards later. This is not like the RL setting, where there is no way for trainers to give feedback ruling out the "whatever the user will reward is good" hypothesis. The user can and should give feedback against this hypothesis!

The AI system accepts all previous feedback, but it may or may not trust anticipated future feedback. In particular, it should be trained not to trust feedback it would get by manipulating humans (so that it doesn't see itself as having an incentive to manipulate humans to give specific sorts of feedback).

I will call this property of feedback "legitimacy". The AI has a notion of when feedback is legitimate, and it needs to work to kee...
Levers for Biological Progress - A Response to "Machines of Loving Grace"
by Niko_McCarty — 2024-11-01T16:35:08.221Z
Link: https://www.asimov.press/p/levers

By Niko McCarty for Asimov Press

Dario Amodei, the CEO of

Anthropic

, recently published an essay called “

Machines of Loving Grace

.” It sketches out his vision for how AI could radically transform neuroscience, economics, diplomacy, and the meaning of work. Amodei also imagines the ways AI could accelerate biological research and yield miraculous cures in the 21st century; everything from the prevention and treatment of nearly all infectious and inherited diseases to the elimination of most cancers.

“Biology is probably the area where scientific progress has the greatest potential to directly and unambiguously improve the quality of human life,” Amodei writes. “My basic prediction is that AI-enabled biology and medicine will allow us to compress the progress that human biologists would have achieved over the next 50-100 years into 5-10 years.”

This is an inspiring vision, but as Amodei acknowledges, achieving it will first require that we think deeply about existing bottlenecks and then roadmap ways to solve them. Indeed, most of what we publish at Asimov Press are roadmaps of this kind, including essays that examine persistent obstacles to scientific progress (such as “

Where’s the Synthetic Blood?

”) or speculative fiction that imagines possible futures once these obstacles have been overcome (see “

Tinker

” or “

Models of Life

”).

Amodei’s essay considers what might happen to biological research should a “powerful AI” emerge that is “smarter than a Nobel Prize winner.” However, it isn’t clear that such a superintelligence could even be applied to its full potential in biology today, given the dearth of high-quality datasets needed to train it.

Although Amodei does acknowledge some real-world issues limiting scientific progress — such as the slow growth of organisms and tedious clinical trials — he mostly passes over the more general tools that will be required to accelerate research in the near term. Still, many of the bottlenecks slowing biology today are biophysical, rather than computational. Therefore, I’m using Amodei’s essay as a rallying cry for researchers to innovate their way past existing bottlenecks in wet-lab biology, which, if achieved, would help scientists actually build more powerful AI models in the future.

It wasn’t easy for me to write this essay because it’s often difficult to predict exactly where a solution to a given problem will emerge. ...
2024 Unofficial LW Community Census, Request for Comments
by Screwtape — 2024-11-01T16:34:14.758Z

Overview

The LessWrong Community Census is an entertaining site tradition that doubles as a useful way to answer various questions about what the userbase looks like. This is a request for comments, constructive criticism, careful consideration, and silly jokes on the census.

Here's the draft.

I'm posting this request for comments on November 1st. I'm planning to incorporate feedback throughout November, then on December 1st I'll update the census to remove the "DO NOT TAKE" warning at the top, and make a new post asking people to take the census. I plan to let it run throughout all December, close it in the first few days of January, and then get the public data and analysis out sometime in late January.

How Was The Draft Composed?

I copied the question set from 2023. Last year, there were a lot of questions and the overall sentiment was that there should be fewer questions, so I removed a lot of things that either didn't seem interesting to me or didn't have much history with the census plus collapsed some of the answer variety that doesn't get used much. This included gutting all but one question in the detailed politics section, though I mean to put a few new ones there. Then I changed some things that change every year like the Calibration question, and swapped around the questions in the Indulging My Curiosity section.

Changes I'm Interested In

No seriously, I want to keep the question count down this year. Right now I think we're a little below 100 (down from ~150 last year) and I plan to keep things under 100. This should be the current arrangement, at 95:

Number	Section	Question Budget 2024
0	Population	3
1	Demographics	5
2	Sex and gender	10
3	Work and education	3
4	Politics	7
5	Intellect	5
6	LessWrong Basics	7
7	LessWrong Community	7
8	Probability	15
9	Traditional	5
10	LW Team	5
11	Adjacent Communities	5
12	My Curiosity	5
13	Detailed past questions	5
14	Bonus Politics	5
15	Wrapup	3

I currently have zero actual questions in the Questions From Adjacent Communities section. Ideally I'd like to get, say, a question from the Forecasting community, a question from the Glowfic community, a question from EA, etc, and add up to 5 questions there. I'll be actively reaching out to organizers and managers of those groups, but if anyone wants proactively step forward in the comments please do!

I currently have only one question in Bonus Politics. I don't find politics interesting, lots of people do, so here's an o...