trentmkelly/LessWrong-43k huggingface dataset:
AXRP Episode 38.2 - Jesse Hoogland on Singular Learning Theory
by DanielFilan — 2024-11-27T06:30:03.821Z

YouTube link

You may have heard of singular learning theory, and its “local learning coefficient”, or LLC - but have you heard of the refined LLC? In this episode, I chat with Jesse Hoogland about his work on SLT, and using the refined LLC to find a new circuit in language models.

Topics we discuss:

About Jesse

The Alignment Workshop

About Timaeus

SLT that isn’t developmental interpretability

The refined local learning coefficient

Finding the multigram circuit

Daniel Filan (00:09):
Hello, everyone. This is one of a series of short interviews that I’ve been conducting at the Bay Area Alignment Workshop, which is run by FAR.AI. Links to what we’re discussing, as usual, are in the description. A transcript is, as usual, available at axrp.net. And as usual, if you want to support the podcast, you can do so at patreon.com/axrpodcast. Well, let’s continue to the interview. All right, Jesse, welcome. Thanks for being interviewed.

Jesse Hoogland (00:32):
Thanks for interviewing me.

About Jesse

Daniel Filan (00:34):
Yeah. So for people who don’t know, could you say a little bit about yourself, who you are?

Jesse Hoogland (00:38):
So I’m the Executive Director of Timaeus. We’re a research organization working on applications of singular learning theory. I think we’ll get into more of the details, but concretely, SLT is a theory of Bayesian statistics that answers some questions around generalization - why neural networks are able to generalize as well as they do - and therefore paves the road to applications for evals. Can you understand when your evaluation benchmark is actually going to be predictive about behavior downstream in deployment? There are applications for interpretability: questions like “can you detect that the model is planning to execute a treacherous turn?” It’s the kind of question I hope to answer someday with a better-developed theory of SLT and associated tools for probing these questions.

Daniel Filan (01:26):
Gotcha. And if people are interested in that, we had a previous episode with Dan Murfet on, I think it’s just called Singular Learning Theory, that you all can listen to.

Jesse Hoogland (01:37):
So Daniel Murfet… I mean, he’s the one who really put forth this agenda of applying SLT to alignment, and we’re working very closely together with Dan to make that a reality.

The Aligment Workshop

Daniel Filan (01:49):
Great. So befo...
Hierarchical Agency: A Missing Piece in AI Alignment
by Jan_Kulveit — 2024-11-27T05:49:04.241Z

The most ambitious direction I've worked on in the past few years is a theory of hierarchical agency (as mentioned in the ACS announcement). Recently, I've become worried that the inferential distance between "the median of current AI safety discourse" and "what I and my colleagues at ACS work on" has become too large. After multiple attempts to write a good introduction to hierarchical agency ended up in perpetual draft form, I decided to try something else: explain it to Claude.
This is a lightly edited transcript of the first part of our conversation.
Claude
Shall we begin? Could you start by explaining what you mean by "hierarchical agency" and why you think it's important for AI safety?

Jan
Broadly, there is a pattern in the world where you can often see agents composed of other agents. Like, corporations and their departments, states and citiziens, and so on.

Claude
So you're looking at nested or layered structures of agency - where agents can contain or be made up of other agents. Let me test my understanding with some examples and their properties:

A corporation as an agent that can make decisions and take actions, but it's composed of departments (also agents) which are in turn composed of individual human agents?
States acting as agents while being composed of various bureaucracies and ultimately individual human officials and citizens?

Would biological examples also fit this pattern? Like how a human body could be seen as an agent composed of organs (which have their own regulatory systems) and cells (which make their own decisions about things like protein production)? I'm curious about what makes this perspective particularly relevant for AI safety

Jan
I hope to get to AI safety relevance later.
Yes, biological examples fit well - particularly multicellular organisms, which demonstrate emergent agency at multiple scales. To better define the boundaries of hierarchical agency, it might help to examine some edge cases and non-examples.

Claude
Let me probe the boundaries systematically by analyzing different types of systems:

A crowd of people in a mall - they're a collection of agents, but are they a cohesive higher-level agent? My guess is no, because there's no real coordination or unified decision-making?
A computer network - it's a connected system of parts, but do the individual computers act as agents in a meaningful way? And do

...
Facets and Social Networks
by jefftk — 2024-11-27T03:40:08.689Z

It's rare that an author's interests will fully overlap with any given
reader's. You can choose to post anywhere on a spectrum between:

Narrow: pick a topic and make it your beat. People who care
about that can follow you and pretty reliably see what they're looking
for.

Broad: write about whatever you like. Some readers will be
willing to scroll past posts on uninteresting topics, others will
decide the combination is too noisy to be worth it to them.

Current social network technology strongly favors the former: if you
write about just one area it's much easier for algorithms to figure
out who to show your posts to. And so you see a lot of advice to build
your personal brand about an area: write about cooking, or housing
policy, or military history.

This is very much not for me. I want to write about whatever I want
to write about, which is a lot of different things, and I've generally
just accepted that this is a bad fit for Facebook and the other places
people read my writing. But two exceptions:

Julia and I have a group
for kid stuff, because it seems to be especially polarizing.

With RSS, I (and others) have single-topic
feeds.

If this were just a me problem then it wouldn't be too bad, but this
seems pretty big to me. In trying out
Bluesky I'm running into a bunch of accounts that post about a
range of things I find variously interesting. I can decide whether
they're worth it overall, but this is not the right choice for the
technology to be forcing me to make.

Instead, I've long
wanted a social network built around the idea that each person's
identity and interests have many facets, and tries to match specific
posts with the people that would be interested in them.

I wonder if, at a time when advances in AI are making this kind of
classification problem easier and there's more social networking
competition than there has been for a while, someone might want to
take this on? Perhaps Bluesky's custom
feeds would be a good way to play with this?

(I don't think this can depend on people tagging their own posts,
because people are generally lazy. But something based on classifying
the post based on its content and who has liked it so far seems pretty
promising.)

Comment via: facebook, mastodon, bluesky
Call for evaluators: Participate in the European AI Office workshop on general-purpose AI models and systemic risks
by Tom DAVID — 2024-11-27T02:54:16.263Z

I am sharing this call from the EU AI Office for organizations involved in evaluation. Please take a close look: among the selection criteria, organizations must be based in Europe, or their leader must be European. If these criteria pose challenges for some of you, feel free to reach out to me at tom@prism-eval.ai. We can explore potential ways to collaborate through PRISM Eval. I believe it’s crucial that we support one another on these complex and impactful issues.

The AI office is collecting contributions from experts to feed into the workshop on general-purpose AI models and systemic risks.

The European AI Office is hosting an online workshop on 13 December 2024 (only for specialists), focusing on the evaluation of general-purpose AI models with systemic risk. This is an opportunity for organisations and research groups to showcase their expertise and contribute to shaping the evaluation ecosystem under the EU AI Act.

The event will bring together leading evaluators and the AI Office to exchange insights on state-of-the-art evaluation methodologies for general-purpose AI models. Selected participants will present their approaches, share best practices, and discuss challenges in assessing systemic risks associated with advanced AI technologies.

This initiative aims to foster collaboration and advance the science of general-purpose AI model evaluations, contributing to the development of robust frameworks for ensuring the safety and trustworthiness of these models.

Call for submissions

The AI Office invites evaluators to submit abstracts of previously published papers on the evaluation of general-purpose AI models with systemic risk. Key topics include:

CBRN Risks: Risks related to chemical, biological, radiological, and nuclear threats
Cyber Offense: Risks associated with offensive cyber capabilities
Major Accidents: Risks of large-scale disruptions or infrastructure interference
Loss of Control: Concerns about oversight and alignment of autonomous AI models
Discrimination: Risks of generating discriminatory outcomes
Privacy Infringements: Risks involving privacy breaches or data misuse
Disinformation: Risks tied to the propagation of false or harmful information
Other Systemic Risks: Additional risks affecting public health, safety, democratic processes, or fundamental rights

Follow the link to take part of the call. Find more information on the application...
Wagering on Will And Worth (Pascals Wager for Free Will and Value)
by Robert Cousineau — 2024-11-27T00:43:08.522Z

Epistemic Status: Quite confident (80%?) about the framework being very useful for the subject of free will. Pretty confident (66%?) about the framework being useful for meta-ethics. Hopeful (33%) that I am using it to bring out directionally true statements about what my CEV would be in worlds where we have yet to have found objective value.

Most discussions about free will and meaning seem to miss what I understand to be the point. Rather than endlessly debating the metaphysics, we should focus on the decision-theoretic implications of our uncertainty. Here's how I[1] think we can do that, using an abstracted Pascal's Wager.

Free Will: A Pointless Debate

People argue endlessly about whether we have Free Will, bringing up quantum mechanics, determinism, compatibilism, blah, blah (blah). But, regardless of if we have it or not:

In worlds where we have no free will:

Our beliefs about free will don't matter (we'll do whatever we were determined to do)
Our beliefs about what our actions should be don't matter and can't be changed (they were predetermined)

In worlds where we have free will:

Our beliefs about free will will affect what we will do (and what we will will change what will happen) (possibly to a Will) (unless our will won't work)
Our choices compound through causality, affecting countless other conscious creatures

Therefore, if we have free will, believing in it and acting accordingly is incredibly valuable. If we don't have free will, nothing we (choose to) believe matters anyway. The expected value clearly points towards acting as if we have free will, even if we assign it a very low probability (I don't think too much about what numbers should be here[2] but estimate it at 5-20%).

Meaning and All that is Valuable:

I've found I am unfortunately sympathetic to some nihilistic arguments:
Whether through personal passing, civilizational collapse, or the heat death of the universe, all information about our subjective experiences, be they bliss, bitterness, fulfillment, or failure, will eventually be lost.

Further, even if it truly is just about living in the moment, that which we as humans value - our goals, our emotions, our moral intuitions - are merely mesa-optimizations that contributed to reproductive fitness in our ancestral environment. It would be remarkably convenient if these specific things happened to be what is a...
Should you have children? All LessWrong posts about the topic
by Sherrinford — 2024-11-26T23:52:44.113Z

Currently, there are 26 LessWrong forums posts tagged "family planning", the oldest from 2010. For a writing project, I read all of them. However, I realized that this collection may interest other people, so I publish it as a stand-alone post. I summarize the posts and their comments in the following. Feedback and comments are welcome.

2010/2011

In "rationality and being child-free" (20th Nov 2010), InquilineKea asks "So how do you think being child-free relates to rationality/happiness?" In the comments, some people discuss the effect of having children on parental happiness and life quality, or state their personal preferences about having children.

The arguments against children include the time-intensity of having children (and the personal preference for autonomy about your time usage), emotional aversion and feelings "unprepared".
Arguments on the pro-side include: Liking children, having children is seen as a public-good contribution, and people discuss whether rationalists should have kids to spread their culture (or whether children are desirable memecarriers for their parents), and some kind of selective pro-natalism according to which the "the future world would likely be a better place if wealthy, educated and responsible people have more kids."
Both on the contra-side and on the pro-side, it is noted that emotions (like insecurity) or desire are very powerful in determining decisions. It is also noted that this can lead to justify an emotion, urge or decision.
It is also told how there can be strong social pressure against having children in a certain milieu (in the 1980s) that both came from seeing children (or rather, people) as a negative factor in the world, but also "the world to be too horrible to bring children into." The counter-position to both claims is then mentioned (the author sees his own children as making the world better and the world to be better than ever).

In June 2011, InquilineKea considers "Mentoring as an alternative to having children?" as this yields "many of the benefits and few of the costs" but notes that there may be "something psychological missing out from all this". In the comments, people discuss other alternatives like sperm donations.

2012/2013

jsalvatier's Review: Selfish Reasons to Have More Kids (May 2012) summarizes the book which notes that the costs of having kids are lower than you think because you don't

...
Dave Kasten's AGI-by-2027 vignette
by davekasten — 2024-11-26T23:20:47.212Z

(Epistemic status: Very loosely held and generated in a 90-minute workshop led by @Daniel Kokotajlo, @Thomas Larsen, @elifland, and Jonas Vollmer at The Curve Conference; explores how it might happen, if it happens soon. I expect there to be at least one "duh, that makes no sense" discovered with any significant level of attention that would require me to rethink some of this.)

Recently, at The Curve conference, I participated in a session that helped facilitate us writing AGI vignettes -- narratives of how we get to AGI. [1] I got feedback that folks thought it was a good vignette, so I'm publishing it here.

My scenario for Stage 1: runup to AGI

AGI is created in early- to mid-2027. It just

works first time

.

Anthropic does it. In many worlds, it could have been OpenAI or Google or Meta or even small startups that figured out One Weird Trick, but in this world, it’s Anthropic. It’s a close-run race; in the months ahead of it, everyone knew each of the frontier labs were nearing in on the finish line; some of them released models or new features that were damnsome close, but Anthropic just got there first. Ultimately, the writer of this narrative is a recovering org practice consultant from McKinsey that thinks org culture matters a lot, and so in this world the crucial factor was that Anthropic staff just had the psychological safety [2]to navigate the terrifying go/no-go decisions on the last few development and deployment phases a little bit better and faster than the others. (In retrospect, that’s why Anthropic started pulling ahead in late 2024, though the lead was far too narrow to call definitive or clear at that point).

Anthropic calls it Shannon.

The others were close, though. Google was probably 9-12 months away, despite all of their constant organizational chaos, because they got their shit together after their first down quarter led to a legendary screaming match at the all-hands. Meta had slowed down after the Trump administration started asking hard questions about open AI weights getting shared with China, and the organizational tax of spinning up all of the APIs and hosting for their newly-closed models, as well as stapling CoT on top of Llama (they’d not wanted to take the inference-time hit back when the model was still open, especially since th...
Fractals to Quasiparticles
by James Camacho — 2024-11-26T20:19:29.675Z

The Serpinski triangle is usually introduced as a fractal made up of three smaller Serpinski triangles. However, that isn't enough to know what it looks like. Which of these is the Serpinski triangle?

The issue with recursive definitions is they create a vicious circle. To get around this, you need to specify a base case, and constructively build out your definition from there. The Python code used to generate the middle triangle looks somewhat like

import turtle
def serpinski(length, min_length=5):
if length < min_length:
return # Stop recursing!

for i in range(3):
serpinski(length / 2, min_length=min_length)
turtle.forward(length)
turtle.left(120)

serpinski(400)

Notice that it halts the recursion after a minimum length. Even if it were not hardcoded, Python's builtin recursion limit or the computer's hardware would force the base case with a RecursionError or MemoryError. Unfortunately, this also means the triangles are no longer self-similar. The outer triangle has six subdivisions, and is made out of triangles with five subdivisions. This can be rectified with an axiom of infinity; then we can choose each triangle to have an ordinal number ω of subdivisions, and since ω=−1+ω it will be self-similar. However, at the end of the subdivisions we still need to define a base case!

Ordinal numbers are sometimes represented by bars, so

1=|,2=||,n=n bars|||…|,ω=|||⋯,

and the Serpinski triangle looks somewhat like

|||⋯△.

Space too is self-similar, so it should have a base defined. In algebraic geometry, these are called simplices. Below, we have a 0-simplex, 1-simplex, and 2-simplex. An n-simplex exists in n-dimensional space, and we build lines, shapes, or volumes by chaining together simplices.

Notice that the boundary of each simplex is built from simplices of one fewer dimension (shown above with arrows). In our notation, [012…n] will mean an n-simplex, and ∂ the boundary operator. We can explicitly calculate the boundary by projecting (dropping) one dimension at a time,

∂[012…n]=n∑k=0(−1)k[01…k−1,k+1…n].

The factor (−1)k makes it so simplices chain together nicely:

We find

∂([012]+[132])=[12]−[02]+[01]+[32]−[12]+[13]=[01]+[13]+[32]−[02]=[01]+[13]+[32]+[20](since −∂[02]=∂[20])

i.e. a path from 0→1→3→2→0. The shared boundary gets...
What epsilon do you subtract from "certainty" in your own probability estimates?
by Dagon — 2024-11-26T19:13:46.795Z

Ok, nobody is actually a strict, or even particularly careful bayesean reasoner. Still, what probability do you reserve to "my model doesn't apply, everything I know is wrong"? If you SEE a coin flip come up heads (and examine the coin and perform whatever tests you like), what's your posterior probability that the coin actually exists and it wasn't a false memory or trick in some way?
Implications—How Conscious Significance Could Inform Our lives
by James Stephen Brown — 2024-11-26T17:42:49.085Z

What is Conscious Significance

In

a case for Conscious Significance

, I outlined a framework for understanding our sense of autonomy, distinct from libertarian ‘free will’, that is determinism-agnostic and yet maintains all the important aspects of our conscious experience of autonomy. This perspective has significant overlap with Eliezer Yudkowsky's Requiredism. I clarified that acting with an acknowledgment of ‘conscious significance’ is not the same as acting we have free will-rather it is acting inline with an understanding that our conscious effort is significant to determining outcomes. At the end of that piece I promised to outline what ethical consequences there are for ‘conscious significance’.

Paradigm Shifts

This began with an exploration of

paradigm shifts

and how they change less than we expect, which warned that premature conclusions which contradict cultural evolution may be disastrous. In the case of determinism, if one takes a naive interpretation that they can do whatever they want because they have no other choice, then they may cause long-term misery for themselves and others.

Some Final Throat-Clearing

The following is an exploration of what does and doesn’t change in a world that might be deterministic, but where many of our norms carry an assumption of ‘free will’. ‘Conscious significance’ distinct from ‘determinism’ and ‘free will’ has the capacity to maintain scientific integrity, without abandoning important norms. It has implications for how we deal with the extremes of behaviour, regarding prison reform, how we frame our internal lives-differentiating guilt and shame, and it can also inform our politics.

So, What Remains Unchanged?

While the position I’m putting forward is determinism-agnostic, for the sake of argument, I’m going to presume determinism, as this is the hardest position against which to test the implications of ‘conscious significance’.

Consequences

What could be more profound than finding out that none of what you do is your fault, but rather the inevitable consequence of a series of zillions of quantum operations? It seems to take us off the hook for a lot, right? But does this mean we can start lazying around doing nothing or even start committing crimes without consequence?

Not so fast…

None of this undermines consequences; in fact, determinism guaranteesconsequences and makes them somewhat foreseeable. It is precisel...