trentmkelly/LessWrong-43k huggingface dataset:
The Case for Predictive Models
by Rubi J. Hudson — 2024-04-03T18:22:20.243Z

Thanks to Johannes Treutlein and Paul Colognese for feedback on this post.

Just over a year ago, the

Conditioning Predictive Models

paper was released. It laid out an argument and a plan for using powerful predictive models to reduce existential risk from AI, and outlined some foreseeable challenges to doing so. At the time, I saw the pieces of a plan for alignment start sliding together, and I was excited to get started on follow-up work.

Reactions to the paper were mostly positive, but discussion was minimal and the ideas largely failed to gain traction. I suspect that muted reception was in part due to the size of the paper, which tried to both establish the research area (predictive models) and develop a novel contribution (conditioning them). Now, despite retaining optimism about the approach, even the authors have mostly shifted their focus to other areas.

I was recently in a conversation with another alignment researcher who expressed surprise that I was still working on predictive models. Without a champion, predictive models might appear to be just another entry on the list of failed alignment approaches. To my mind, however, the arguments for working on them are as strong as they’ve ever been.

This post is my belated attempt at an accessible introduction to predictive models, but it’s also a statement of confidence in their usefulness. I believe the world would be safer if we can reach the point where the alignment teams at major AI labs consider the predictive models approach among their options, and alignment researchers have made conscious decisions whether or not to work on them.

What is a predictive model?

Now the first question you might have about predictive models is: what the heck do I mean by “predictive model”? Is that just a model that makes predictions? And my answer to that question would be “basically, yeah”.

The term predictive model is referring to the class of AI models that take in a snapshot of the world as input, and based on their understanding output a probability distribution over future snapshots. It can be helpful to think of these snapshots as represented by a series of tokens, since that’s typical for current models.

As you are probably already aware, the world is fairly big. That makes it difficult to include all the information about the world in a model’s input or output. Rather, predictive models need to work with more limit...
Book Review (mini): Co-Intelligence by Ethan Mollick
by Darren McKee — 2024-04-03T17:33:52.097Z

A brief and pleasant exploration of how AI is shaping the world around us, with tips and suggestions for educators, workers, and students to use AI as a Coworker, Tutor, Coach, or another Person. It is best seen as a business book or read by those with little AI exposure.

Admittedly, given how much I enjoy Mollick's tweets about AI, I was hoping for a bit more (but it is a general audience business book, so I understand).

I appreciated the refrain that the AI model you're using now will be the worst one you ever use. Additionally, at the very end, he briefly mentions four possibilities for the future:

1. No AI growth

2. Slow AI growth

3. Exponential AI growth

4. Machine God (end of human dominance)

I think it is useful to describe the main futures we are likely to inhabit, so this was a welcome addition. That said, he very much does not focus on AI as a larger threat to humanity. I think this is fine because a book can't be about everything, but his reasons for avoiding the topic were not compelling. Paraphrasing, he said that larger existential concerns have a lot of uncertainty, and if there was an issue, it's quite disempowering. It's that last part that is the issue. Discussing AI risk doesn't have to be disempowering, it could be galvanizing. We can actually get people to become engaged and be more concerned about AI safety. Mollick seems to somewhat believe this because he urges his audience to become more engaged on (less dramatic) AI issues lest their future be decided for them. The same companies and complications are involved (to a large extent, anyway) in both forms of advocacy, so I don't know why he codes them differently.

If someone wrote a book about air pollution and wildfires, that's perfectly fine. But if they said they weren't going to discuss climate change because it would be disempowering, that would be an odd move.

Finally, I sympathize with the difficulties of narrating an audiobook and the inclination to have the author read their book, but sorry to say that the listening experience would have been better with another narrator.
Sparsify: A mechanistic interpretability research agenda
by Lee Sharkey — 2024-04-03T12:34:12.043Z

Over the last couple of years, mechanistic interpretability has seen substantial progress. Part of this progress has been enabled by the identification of superposition as a key barrier to understanding neural networks (

Elhage et al., 2022

) and the identification of sparse autoencoders as a solution to superposition (

Sharkey et al., 2022

;

Cunningham et al., 2023

;

Bricken et al., 2023

).

From our current vantage point, I think there’s a relatively clear roadmap toward a world where mechanistic interpretability is useful for safety. This post outlines my views on what progress in mechanistic interpretability looks like and what I think is achievable by the field in the next 2+ years. It represents a rough outline of what I plan to work on in the near future.

My thinking and work is, of course, very heavily inspired by the work of Chris Olah, other Anthropic researchers, and other early mechanistic interpretability researchers. In addition to sharing some personal takes, this article brings together - in one place - various goals and ideas that are already floating around the community. It proposes a concrete potential path for how we might get from where we are today in mechanistic interpretability to a world where we can meaningfully use it to improve AI safety.

Key frameworks for understanding the agenda

Framework 1: The three steps of mechanistic interpretability

I think of mechanistic interpretability in terms of three steps:

Figure 1: The three steps of Mechanistic Interpretability

The three steps of mechanistic interpretability[1]:

Mathematical description: In the first step, we break the neural network into constituent parts, where the parts are simply unlabelled mathematical objects. These may be e.g. neurons, polytopes, circuits, feature directions (identified using SVD/NMF/SAEs), individual parameters, singular vectors of the weight matrices, or other subcomponents of a network.
Semantic description: Next, we generate semantic interpretations of the mathematical object (e.g. through feature labeling). In other words, we try to build a conceptual model of what each component of the network does.
Validation: We need to validate our explanations to ensure they make good predictions about network behavior. For instance, we should be able to predict that ablating a feature with a purported ‘meaning’ (such as the 'noun gen

...
Just because 2 things are opposites, doesn't mean they're just the same but flipped
by Alok Singh — 2024-04-03T08:59:43.667Z
Link: https://alok.github.io/2024/03/29/broken-dualities/

The 2 Aspects

There’s 2 Aspects to things in general. I will call them Mapping Out and Mapping In, in titlecase so you know they’re distinct concepts.

warmup: 0 -> 1

Here, 0 is an initial object and 1 is a terminal object. 0 is Mapped Out of because it’s 0 -> and not -> 0. 1 is Mapped Into because it’s -> 1 and not 1 ->. The defining property of an initial object is that it has 1 map to every other object. The defining property of a terminal object (1) is that it has 1 map from every other object.

Broken Symmetries

Injective functions have inverses. But the dual statement, that surjective functions always have (at least 1) inverse, is equivalent to the Axiom of Choice!
a AND (NOT a) = false is just the law of noncontradiction. A simple theorem (unless you’re Graham Priest), but the dual statement a OR (NOT a) = true is the Law of Excluded Middle, which can be derived from a form of the Axiom of Choice, and in particular is not constructively valid.
Freyd’s Adjoint Functor Theorem sometimes fails to have a left adjoint (Mapping Out), but for foundations-of-mathematics-y reasons: If the left adjoint existed, it would be too big to be a set. No such difficulty for right.
Googling epimorphism too large to form set gives a lot of stuff. Nothing similar for monomorphism too large to form set

The following is from Paul Garrett, Section 2:

When we look at colimits and coproducts here, it is important to see that, while at an abstract level these things are just the arrows-reversed versions of limits and products, for many classes of naturally-occurring objects there is a sharp asymmetry. For example, while limits are subobjects of products, colimits are quotients of coproducts. In many situations, quotients are more abstract entities than are subobjects. This can be explained from a set-theoretic viewpoint, since elements of a subset are the same sort of thing as elements of the original set, since they are elements of the original set, while elements of quotients are sets of elements of the original.

In particular, in many cases colimits are fragile, and need further details or hypotheses to give us helpful outcomes. For example, while all subspaces of Hausdorff topological spaces are Hausdorff, quotients of Hausdorff topological spaces need not be Hausdorff.

submodules of finitely-generated free modules over principal ideal domains are still free, while quotien

...
Falling fertility explanations and Israel
by Yair Halberstadt — 2024-04-03T03:27:38.564Z

From Robin Hanson, via TheZvi:

The following 8 social trends plausibly contribute to falling fertility:

More gender equality - More equal gender norms, options, & expectations, have contributed to fewer women having kids.

Higher parenting effort - Expectations for how much attention and effort parents give each kid have risen.

Long stiff career paths - The path of school & early career prep til one is established worker is longer & less flexible.

Cap- vs cornerstone marry - Now marrying/kids wait until we fully formed, career established, then find matching mate.

Grandparent less involved - Parents once helped kids choose mates, & helped them raise kids. Now kids more on own.

More urban less rural - People now love in denser urban areas where housing costs more, kids have less space.

Less fundamental religion - Religion once clearly promoted fertility, but we less religious, especially re fundamentalism.

Integrated world culture - We pay less attention to local, and more to global, community comparisons and norms.

Israel is the only developed country with a high TFR, and it's TFR has remained more or less stable at ~3 for the last 30 years or so. It's a good test case to see which of these factors might matter more or less than the others.

Israeli TFR is stratified by religious observance, with more religious people having more kids, but even secular Israelis are at replacement TFR - higher than the average for almost any other developed country:

So let's see how these trends apply to Israel:

More gender equality: gender norms in Israel outside the ultra orthodox community are pretty similar to what you might find in any other developed country. Traditional gender roles are maintained in the ultra orthodox community, but since many of the men study Torah all day, women are often the main bread winners, and expected to look after the kids on top of that.

Higher parenting effort: Anecdotally Israelis make for interesting parents. They are incredibly and obsessively protective of babies and toddlers (and continuously pester me as someone from England that my child too cold, or not drinking enough, or might choke on that food, or whatever), and mostly blasé about any kid above about 4 or 5, at which point they can go to the park by themselves, go shopping by themselves, and pretty much do whatever they like.

Long stiff caree

...
Nature is an infinite sphere whose center is everywhere and circumference is nowhere
by Alok Singh — 2024-04-03T02:24:55.016Z

Blaise Pascal said that. When I heard it, this interpretation instantly came.

Here's an AR companion video where I point at some mountains in the Apple Vision Pro to explain the big idea.

Whether or not the universe is actually infinite, it's real big. So modeling it as hyperfinite is legit[1].

Imagine a Sphere with unlimited radius R>N. TLDR, R is a positive, infinitely big whole number. R>1,R>2…R>n… for all finite n. These pictures are circles instead of spheres, but the analysis carries over unchanged.

Center is everywhere

Imagine moving the center point to any other FINITE point. Here's how the picture would change

It wouldn't.

The point is that removing a finite piece from an infinite thing only affects its size by an infinitesimal amount, a point.

Keep this example in mind: the stars seem to be uniformly far but obviously are not. The abstraction of the Celestial Sphere is based on this. Even though all the stars are drawn on this boundary, they are different (big but finite) distances away. Even though light years separate them, you can’t tell.

Imagine shifting the center in the picture along the radius by a finite amount r (say the distance to the sun, or just watch this) but keep the boundary fixed. Then the length of the new radius is R−r. But that's almost as long as the original radius. neworiginal=R−rR=1−1R≈1. So moving the center is unobservable. So, it's everywhere because you couldn't ever observe that some radii are shorter than others without moving infinitely far to check, which is impossible. You're not truncating the sphere so much as shaving it.

Circumference is nowhere

It's out at R in the Positive Infinite segment, a hyperfinite and therefore inassignable number[2].

Because it's necessarily vague, it's "nowhere" because it cannot be assigned a definite value.

More intuitively, it's really really far away and from your finite portion you can't ever reach it, so is it really there? Does stuff outside the lightcone actually exist? [3]And R is even bigger than that!

Center is nowhere ≈> Circumference is nowhere

The ≈> means 'generically implies'.

Saying the 'center is nowhere' would put it at some infinite coordinate O. Which you can do, but the...
The Rationalist Haggadot Collection
by maia — 2024-04-02T20:02:20.785Z
Link: https://tigrennatenn.neocities.org/rationalist_haggadot/

Passover is coming, which means some of us will be celebrating Secular Seders. For those wanting to celebrate but looking for resources, I present: the Rationalist Haggadot Collection, an archive of all rationalist Seder ritual books known to me. Pick and choose whatever parts you like best from these, or just pick your favorite.

If you or someone you know uses a different haggadah for a Secular Seder that isn't already listed here, please send it to me! I want to make sure these aren't lost.
How Often Does ¬Correlation ⇏ ¬Causation?
by niplav — 2024-04-02T17:58:41.695Z

Current best guess: Nearly all the
time55%.

"Correlation ⇏ Causation" is trite by now. And we also know that
the
contrapositive
is false too:
"¬Correlation ⇏ ¬Causation".

Spencer Greenberg
summarizes:

All of this being said, while causation does not NECESSARILY imply
correlation, causation USUALLY DOES imply correlation. Some software
that attempts to discover causation in observational data even goes so
far as to make this assumption of causation implying correlation.

I, however, have an inner computer scientist.

And he demands answers.

He will not rest until he knows how often ¬Correlation ⇒ ¬Causation,
and how often it doesn't.

This can be tested by creating a Monte-Carlo
simulation
over random linear structural equation
models
with n variables, computing the correlations between the different
variables for random inputs, and checking whether the correlations being
zero implies that there is no causation.

So we start by generating a random linear SEM with n variables (code
in Julia). The parameters are normally distributed with mean 0 and variance 1.

function generate_random_linear_sem(n::Int)
g = DiGraph(n)
for i in 1:n
for j in (i+1):n
if rand() < 0.5
add_edge!(g, i, j)
end
end
end
coefficients = Dict()
for edge in edges(g)
coefficients[edge] = randn()
end
return g, coefficients
end

We can then run a bunch of inputs through that model, and compute their
correlations:

function correlation_in_sem(sem::DiGraph, coefficients::Dict, inner_samples::Int)
n = size(vertices(sem), 1)
input_nodes = [node for node in vertices(sem) if indegree(sem, node) == 0]
results = Matrix{Float64}(undef, inner_samples, n) # Preallocate results matrix
for i in 1:inner_samples
input_values = Dict([node => randn() for node in input_nodes])
sem_values=calculate_sem_values(sem, coefficients, input_values)
sem_value_row = reshape(collect(values(sort(sem_values))), 1, :)
results[i, :] = sem_value_row
end
correlations=cor(results)
for i in 1:size(correlations, 1)
correlations[i, i] = 0
end
return abs.(correlations)
end

We can then check how many correlations are "incorrectly small".

Let's take all the correlations between variables which don't have
any causal relationship. The largest of those is the "largest uncaused...
[EA xpost] The Rationale-Shaped Hole At The Heart Of Forecasting
by dschwarz — 2024-04-02T17:40:44.278Z
Link: https://forum.effectivealtruism.org/posts/qMP7LcCBFBEtuA3kL/the-rationale-shaped-hole-at-the-heart-of-forecasting

An excerpt from the above that will be relevant to this crowd:
Ben Landau-Taylor of

Bismarck Analysis

wrote a piece on March 6 called “

Probability Is Not A Substitute For Reasoning

”, citing a piece where he writes:

There has been a great deal of research on what criteria must be met for forecasting aggregations to be useful, and as Karger, Atanasov, and Tetlock

argue

, predictions of events such as the arrival of AGI are a very long way from fulfilling them.

Last summer,

Tyler Cowen wrote on AGI ruin forecasts

:

Publish, publish, not on blogs, not long stacked arguments or six hour podcasts or tweet storms, no, rather peer review, peer review, peer review, and yes with models too... if you wish to convince your audience of one of the most radical conclusions of all time…well, more is needed than just a lot of vertically stacked arguments.

Widely divergent views and forecasts on AGI persist, leading to FRI’s excellent

adversarial collaboration on forecasting AI risk

this month. Reading it, I saw… a lot of vertically stacked arguments.

<...>
Tyler Cowen

again

:

If the chance of existential risk from AGI is 99 percent, or 80 percent, or even 30 percent, surely some kind of modeled demonstration of the basic mechanics and interlocking pieces is possible.

It is possible! It’s much harder than modeling geopolitics, where the future more resembles the past. I’m partial to Nuño’s

base rates of technological disruption

which led him to posit “30% that AI will undergo a ‘large and robust’ discontinuity, at the rate of maybe 2% per year if it does so.” The beauty of his analysis is that you can inspect it. I think Nuño and I would converge, or get close to it, if we hashed it out.

Other great examples include

Tom Davidson’s compute-centric model

,

Roodman's “materialist” model

, and Joe Carlsmith’s

six ingredients model

. These models are full of prose, yet unlike pure reasoning, they have facts you can substitute and numbers you can adjust that directly change the conclusion.

I bet that if the FRI adversarial collaborators had drawn from Sempere’s, Davidson’s, Roodman’s, or Carlsmith’s models, they would have converged more. A quick ctrl+f of the 150 page FRI report shows only two such references - both to Davidson’s... appearance on a podcast! The 2022 GJ report used the Carlsmith model to generate the questions, but it appears none ...
Religion = Cult + Culture
by Eneasz — 2024-04-02T16:44:27.010Z
Link: https://deathisbad.substack.com/p/religion-cult-culture

[copied in full -- request to develop community knowledge/practices?]

Cults are not necessarily bad. Cults provide value. People join them to get things they need which aren’t provided elsewhere. Every cult is a spiritual start-up, doing its best to serve a neglected segment of the population.

Start-ups are famous for the intensity of focus and commitment they inspire in their founding cohort. Observations from Paul Graham:

Running a startup is not like having a job or being a student, because it never stops. This is so foreign to most people's experience that they don't get it till it happens.

I didn't realize I would spend almost every waking moment either working or thinking about our startup. You enter a whole different way of life when it's your company vs. working for someone else's company.

It's surprising how much you become consumed by your startup, in that you think about it day and night, but never once does it feel like "work."

Start-ups are unsustainable. The amount of work, focus, and stress they require always brings people to burn-out eventually. The runway of a start-up isn’t just measured in the money it needs to sustain itself and grow, it’s also measured in how many dozens of months its initial cohort can work this intensely before collapsing. The point of every start-up is to create something that can transition into a stable company before those resources run out.

Cults have a start-up culture. Everyone within them is excited and in love with their work and can focus on little else, and it’s great. But it is equally unsustainable. A cult is inspirational and fulfilling, but it doesn’t interface well with the wider world. The demands and pressures of real life stack higher and higher until eventually something breaks.

For a cult to continue to serve its members for many decades (or centuries), providing value to their children and grandchildren and the surrounding community, it must adopt adopt techniques that allow its members to lead functional lives outside of its confines. It has to interface with the wider world and be shaped by its practicalities.

Phil Goetz described what this looks like in 2009: the culture surrounding a cult turns it into a religion by providing it with memetic antibodies — practices which allow the standard believer to interface normally with the rest of the world.

People who grow up with a religion learn how

...
BIDA Election Thoughts
by jefftk — 2024-04-02T15:30:05.990Z

At this Sunday's dance BIDA will be holding its annual meeting, which
means at the break there will paper ballots for voting on two things:

Board: Who will run the organization for the next year?

Bylaws: Three proposed changes to simplify
elections.

Harris wrote up a blog post with a sample
ballot and candidate statements. Possibly I should have expected
this from our unprecedented
recent attendance, but this is the most people we've ever had
running, by a large margin.

I'm not making any endorsements for the election: I've enjoyed working
with the people who are running for re-election, and I'd be happy
working with any of the people running to join. I'm glad to see this
level of enthusiasm for helping make this all-volunteer organization
happen!

On the bylaw changes, I think all three would be good. I think the
case for (C), switching to standard approval voting, is by far the
strongest: there's no reason to limit how many people you can vote
for. The cases for (A) and (B) are also pretty good: (A) avoids
asking dancers to fill out ballots which don't do anything, which
happened in 2022, and (B) handles a case where the board is
functioning well, has more than seven people, and no one is interested
in leaving or joining. Of these, (A) and (B) would be irrelevant this
year (and I generally prefer them to stay irrelevant—interest in
joining is good!) while (C) would help a lot (if there are a lot of
people you want to vote for and a few people you'd like to vote
against).

Comment via: facebook, mastodon