trentmkelly/LessWrong-43k huggingface dataset:
Atoms to Agents Proto-Lectures
by johnswentworth — 2023-09-22T06:22:05.456Z
Link: https://www.youtube.com/playlist?list=PLJ9u8EFfTLel8wpw4mZClAgVufYx9aAlF

You know the "NAND to Tetris" book/course, where one builds up the whole stack of a computer from low-level building blocks? Imagine if you had that, but rather than going from logic gates, through CPUs and compilers, to a game, you instead start from physics, go through biology and evolution, to human-like minds.

The Atoms to Agents Proto-Lectures are not that. They don't even quite aspire to that. But they aspire to one day aspire to that.

Basically, I sat down with Eli Tyre and spent a day walking through my current best understanding/guesses about the whole agency "stack", both how it works and how it evolved. The result is unpolished, full of guesswork, poorly executed (on my part), and has lots of big holes. But it's also IMO full of interesting models, cool phenomena, and a huge range of material which one rarely sees together. Lots of it is probably wrong, but wrong in ways that illuminate what answers would even look like.

The whole set of proto-lectures is on youtube here; total runtime is about 6 hours, broken across six videos. Below is a rough outline of topics. [EDIT: ATheCoder cleaned up the audio and posted now-better videos here; I've also updated the link at the top of this post to point there. Thankyou!]

Key properties of low-level physics (proto-lecture 1)

Locality
Symmetry
A program-like data structure is natural for representing locality + symmetry

Chaos (proto-lecture 2)

How information is "lost" via chaos
Conserved quantities
Sequences of Markov Blankets as a tool to generalize chaos beyond time-dynamics

Objects (beginning of proto-lecture 3)

What does it mean for two chunks of atoms at two different times to "be the same object" or to "be two copies of the same object"?
What would mean for an object to "copy" over time, in a sense which could ground bio-like evolution in physics?

Abiogenesis and evolution of simple agents (proto-lecture 3, beginning of 4)

Autocatalytic reactions
Membranes/physical boundaries
Complex molecules from standardized parts: RNA world, proteins
Durable & heritable “blueprint”: the genome
Transboundary transport
Internal compartments
Making “actions” a function of “observations”
Bistability -> memory
Consistent trade-offs -> implicit "prices"
Mobility

Multicellularity & Morphogenesis (proto-lecture 4)

Self-assembly at the molecular scale: bulk, tubes, surfaces
Sticky ball
Specialization again
Body axes
Gastrulation: boundarie

...
Would You Work Harder In The Least Convenient Possible World? 
by Firinn — 2023-09-22T05:17:05.148Z

Part one of what will hopefully become the aspirant sequence.

Content note: Possibly a difficult read for some people. You are encouraged to just stop reading the post if you are the kind of person who isn’t going to find it useful. Somewhat intended to be read alongside various more-reassuring posts, some of which it links to, as a counterpoint in dialogue with them. Pushes in a direction along a spectrum, and whether this is good for you will depend on where you currently are on that spectrum. Many thanks to Keller and Ozy for insightful and helpful feedback; all remaining errors are my own.

Alice is a rationalist and Effective Altruist who is extremely motivated to work hard and devote her life to positive impact. She switched away from her dream game-dev career to do higher-impact work instead, she spends her weekends volunteering (editing papers), she only eats the most ethical foods, she never tells lies and she gives 50% of her income away. She even works on AI because she abstractly believes it’s the most important cause, even though it doesn’t really emotionally connect with her the way that global health does. (Or maybe she works on animal rights for principled reasons even though she emotionally dislikes animals, or she works on global health even though she finds AI more fascinating; you can pick whichever version feels more challenging to you.)

Bob is interested in Effective Altruism, but Alice honestly makes him a little nervous. He feels he has some sort of moral obligation to make the world better, but he likes to hope that he’s fulfilled that obligation by giving 10% of his income as a well-paid software dev, because he doesn’t really want to have to give up his Netflix-watching weekends. Thinking about AI makes him feel scared and overwhelmed, so he mostly donates to AMF even though he’s vaguely aware that AI might be more important to him. (Or maybe he donates to AI because he feels it’s fascinating, even though he thinks rationally global health might have more positive impact or more evidence behind it - or he gives to animal rights because animals are cute. Up to you.)

Alice: You know, Bob, you claim to really care about improving the world, but you don’t seem to donate as much as you could or to use your time very effectively. Maybe you should donate that money rather than getting takeout tonight?

Bo...
Contra Kevin Dorst's Rational Polarization
by azsantosk — 2023-09-22T04:28:42.683Z

Epistemic status: trying hard to explain, not persuade. Part of me wants to fight the good fight and protect Bayesian orthodoxy against a corrupting heresy.

MIT Professor Kevin Dorst (LessWrong username: kevin-dorst) has a new argument that, contrary to standard Bayesianism, it can be rational to predict in which directions your beliefs will change.

He explicitly argues against the Conservation of Expected Evidence, and applies this argument to a variety of situations, particularly to political polarization.

His new paper has been generating attention, including from renowed figures such as Steven Pinker, and Dorst has been invited on podcasts to defend his position. Dorst also recently made a frontpage post here on this community, named Polarization is Not (Standard) Bayesian, stating similar claims. These claims deserve to be addressed.

Reflection Violations

Here is one of his examples (from the paper):

Let s be a politically-coded belief, e.g. that guns increase safety. You are currently unsure about s, say 50-50, but if you also know that you're going to study in a liberal college. Students going to liberal colleges are likely to leave with more liberal opinions, and you don't think you are special regarding that effect. He argues that it can be rational to both hold your current belief regarding s, and to expect your future belief on s to move in the liberal direction.

He calls situations like that reflection violations. Here Dorst gives us another example:

Reflection violations are mundane. We can often predict how our actions will shift our beliefs, even when those actions provide no evidence about the issue. Not long ago, I had both Piketty’s Capital in the 21st Century and Pinker’s Enlightenment Now on my shelf. It wasn’t hard to predict that reading Pinker would make me more optimistic about our economic system, and reading Piketty would make me less.

In the LessWrong article, he gives a different example of reflection violations (also called martingale violations), arguing that even if you dont know in which direction your beliefs are likely to move (e.g. in the liberal or conservative direction), there can still be a reflection violation on the correlation of beliefs:

Suppose Nathan is about to immerse himself in political discourse and form some opinions. Currently he’s 50-50 on whether Abortion is wrong (A) and on whether Guns increase safety (G).

...
What social science research do you want to see reanalyzed?
by Michael Wiebe — 2023-09-22T00:03:01.418Z

I've been awarded a small Lightspeed Grant to replicate empirical social science research. What research should I look at?

I'm a PhD economist with an interest in reanalyzing published research using different methods or data (eg. checking whether the results are robust to different regression models, rather than rerunning a lab experiment). I've looked at whether mayors in China are promoted based on GDP growth, the effect of racial violence on patenting, and the effect of medical marijuana legalization on crime. I've also done work on air pollution and mortality, the long-run impacts of the measles vaccine, and how tech clusters drive innovation.
Immortality or death by AGI
by ImmortalityOrDeathByAGI — 2023-09-21T23:59:59.545Z
Link: https://forum.effectivealtruism.org/posts/KkmbSbu2MhLmyj2Rd/immortality-or-death-by-agi

AKA My Most Likely Reason to Die Young is AI X-Risk

TL;DR: I made a model which takes into account AI timelines, the probability of AI going wrong, and probabilities of dying from other causes. I got that the main “end states” for my life are either dying from AGI due to a lack of AI safety (at 35%), or surviving AGI and living to see aging solved (at 43%).

Meta: I'm posting this under a pseudonym because many people I trust had a strong intuition that I shouldn't post under my real name, and I didn't feel like investing the energy to resolve the disagreement. I'd rather people didn't de-anonymize me.

The model & results

A simple model of AGI scenarios. All possible "end" states of the simulation are in color.

I made a simple probabilistic model of the future, which takes seriously the possibility of AGI being invented soon, its risks, and its effects on technological development (particularly in medicine):

Without AGI, people keep dying at historical rates (following US actuarial tables)
At some point, AGI is invented (following Metaculus timelines)
At the point AGI is invented, there are two scenarios (following my estimates of humanity’s odds of survival given AGI at any point in time, which are relatively pessimistic):

We survive AGI.
We don’t survive AGI.

If we survive AGI, there are two scenarios:

We never solve aging (maybe because aging is fundamentally unsolvable or we decide not to solve it).
AGI is used to solve aging.

If AGI is eventually used to solve aging, people keep dying at historical rates until that point.

I model the time between AGI and aging being solved as an exponential distribution with a mean time of 5 years.

Using this model, I ran Monte Carlo simulations to predict the probability of the main end states of my life (as someone born in 2001 who lives in the US):

I die before AGI: 10%
I die from AGI: 35%
I survive AGI but die because we never solve aging: 11%
I survive AGI but die before aging is solved: 1%
I survive AGI and live to witness aging being solved: 43%

Here’s what my model implies for people based on their year of birth, conditioning on them being alive in 2023:

Probabilities of various end states depending on date of birth, my personal estimates for timelines and p(doom).

As is expected, the earlier people are born, the likelier it is that they will die before AGI. The later someone is born, the likelier it is that they will eit...
Neel Nanda on the Mechanistic Interpretability Researcher Mindset
by Michaël Trazzi — 2023-09-21T19:47:02.745Z
Link: https://theinsideview.ai/neel

Some excerpts from my interview with Neel Nanda about how to productively carry out research in mechanistic interpretability.

Posting this here since I believe his advice is relevant for building accurate world models in general.

An Informal Definition Of Mechanistic Interpretability

It's kind of this weird flavor of AI interpretability that says, "Bold hypothesis. Despite the entire edifice of established wisdom and machine learning, saying that these models are bullshit, inscrutable black boxes, I'm going to assume there is some actual structure here. But the structure is not there because the model wants to be interpretable or because it wants to be nice to me. The structure is there because the model learns an algorithm, and the algorithms that are most natural to express in the model's structure and its particular architecture and stack of linear algebra are algorithms that make sense to humans. (context)

Three Modes Of Mechanistic Interpretability Research: Confirming, Red Teaming And Gaining Surface Area

I kind of feel a lot of my research style is dominated by this deep seated conviction that models are comprehensible and that everything is fundamentally kind of obvious and that I should be able to just go inside the model and there should be this internal structure. And so one mode of research is I just have all of these hypotheses and guesses about what's going on. I generate experiment ideas for things that should be true if my hypothesis is true. And I just repeatedly try to confirm it.

Another mode of research is trying to red team and break things, where I have this hypothesis, I do this experiment, I'm like, "oh my God, this is going so well", and then get kind of stressed because I'm concerned that I'm having wishful thinking and I try to break it and falsify it and come up with experiments that would show that actually life is complicated.

A third mode of research is what I call "trying to gain surface area" where I just have a system that I'm pretty confused about. I just don't really know where to get started. Often, I'll just go and do things that I think will get me more information. Just go and plot stuff or follow random things I'm curious about in a fairly undirected fuzzy way. This mode of research has actually been the most productive for me. [...]

You could paraphrase them as, "Isn't it really obvious what's going on?", "Oh man, am I so su

...
Require AGI to be Explainable
by PeterMcCluskey — 2023-09-21T16:11:25.112Z
Link: https://bayesianinvestor.com/blog/index.php/2023/09/20/require-agi-to-be-explainable/

Context: looking for an alternative to a pause on AI development.

There's some popular desire for software decisions to be explainable
when used for decisions such as whether to grant someone a loan. That
desire is not sufficient reason for possibly crippling AI progress. But
in combination with other concerns about AI, it seems promising.

Much of this popular desire likely comes from people who have been (or
expect to be) denied loans, and who want to scapegoat someone or
something to avoid admitting that they look unsafe to lend to because
they've made poor decisions. I normally want to avoid regulations that
are supported by such motives.

Yet an explainability requirement shows some promise at reducing the
risks from rogue AIs.

There are challenges for both developers and regulators in figuring out
what constitutes a useful explanation. It's unlikely that we'll get
guarantees about the accuracy of the explanations, or the enforcibility
of such laws.

I expect that explainability laws, if well written and enforced, will
modestly increase AI transparency, and reduce the risk that AIs will
deceive us about something important.

A need for explainability likely imposes costs on AI development. I
expect it to impose substantially more cost on systems that think in
inhuman concepts than on systems we can hope to understand. That
probably means creating disincentives for developing systems that are
hard to align.

Feasibility

It seems like a common reaction among experts is to dismiss
explainability as unfeasible. That has been my intuitive reaction as
well. But on reflection, I decided I was applying overly perfectionist
standards.

Regulations can be valuable even if compliance falls short of our hopes.

What qualifies as a good explanation? The basic idea could revolve
around picking some inputs, and testing what happens if different values
are provided for those inputs.

A fair amount of research into this topic has assumed that explanations
will be hand-crafted. That seems reasonable for an AI specialized to
approve loans. It is not appropriate for explaining ChatGPT's answers.
I will go through two examples that will hint at how AI's might do most
of the explainability work.

No, that doesn't mean asking the AI in question how it got its answer.
That means empirically testing what conditions cause it to change its
answer. I envision using one or more relatively specialized...
Update to "Dominant Assurance Contract Platform"
by moyamo — 2023-09-21T16:09:57.044Z

Update #2: Platform Launched

This is an update to The Economics of the Asteroid Deflection Problem (Dominant Assurance Contracts)

How the fundraiser went

TL;DR I got $2172.67 dollars even though I only asked for $629.

My prediction of how the fundraiser would go

I expected that I would get ~1000 views on my website, of which 1% would decide to fund me, with an average donation of $90. 1000×1%×$90=$900. I expected that I would get a large initial sum of money and then it would slowly crawl upwards, until getting funded in the last few minutes.

Manifold markets seemed more pessimistic than me (see how on the 28th August there was only a 26% chance I'd raise more than $829) so I lowered the price to $629. This turned out to be unnecessary.

How the fundraiser actually went

After initially posting on Lesswrong, the conversion rate of people visiting https://dac.mowzer.co.za to funding my project was 20%. This was much higher than I expected.

On 2 September, Alex Tabarrok posted my project on marginalrevolution.com (Thanks!). After which the number of visits skyrocketed. The conversion rate lowered to 4%, but this was still higher than the 1% I expected, especially since people kept donating to the project even after it was funded.

After the goal was reached

After the the goal was reached on 2 September, people kept donating! I was not expecting this. I'm really grateful to everyone who donated.

In the end, I got 1300 visits, most from when it was posted on marginal revolution.

What I am going to do

I asked for $639 to work for a month, but since I got more than triple this I'm going to work for 3 months (up to 15 December)!

What I need

I need a name

I'm running a contest on manifold.markets to name my platform. I will PayPal $25 to the person who suggests the winning name

I need producers of public goods

If you are interested in using my platform get funding for something you want to create please fill out this google form.
Sparse Autoencoders: Future Work
by Logan Riggs — 2023-09-21T15:30:47.198Z

Mostly my own writing, except for the 'Better Training Methods' section which was written by @Aidan Ewart.
We made a lot of progress in 4 months working on Sparse Autoencoders, an unsupervised method to scalably find monosemantic features in LLMs, but there's still plenty of work to do. Below I (Logan) give both research ideas, as well as my current, half-baked thoughts on how to pursue them.

Find All the Circuits!

Truth/Deception/Sycophancy/Train-Test distinction/[In-context Learning/internal Optimization]

Find features relevant for these tasks. Do they generalize better than baselines?
For internal optimization, can we narrow this down to a circuit (using something like causal scrubbing) and retarget the search?

Understand RLHF

Find features for preference/reward models that make the reward large or very negative.
Compare features of models before & after RLHF

Adversarial Attacks

What features activate on adversarial attacks? What features feed into those?
Develop adversarial attacks, but only search over dictionary features

Circuits Across Time

Using a model w/ lots of checkpoints like Pythia, we can see feature & circuit formation across time given datapoints.

Circuits Across Scale

Pythia models are trained on the same data, in the same order but range in model sizes from 70M params to 13B.

Turn LLMs into code

Link to very rough draft of the idea I (Logan) wrote in two days

Mechanistic Anomaly Detection

If distribution X has features A,B,C activate, and distribution Y has features B,C,D, you may be able to use this discrete property to get a better ROC curve than strictly continuous methods.
How do the different operationalizations of distance between discrete features compare against each other?

Activation Engineering

Use feature directions found by the dictionary instead of examples. I predict this will generalize better, but would be good to compare against current methods
One open problem is which token in the sequence do you add the vector to. Maybe it makes sense to only add the [female] direction to tokens that are [names]. Dictionary features in previous layers may help you automatically pick the right type e.g. a feature that activates on [names].

Fun Stuff

Othello/Chess/Motor commands - Find features that relate to actions that a model is able to do. Can we find a corner piece feature, a knight feature, a "move here" feature?

Feature Sear

...
Sparse Autoencoders Find Highly Interpretable Directions in Language Models
by Logan Riggs — 2023-09-21T15:30:24.432Z

This is a linkpost for Sparse Autoencoders Find Highly Interpretable Directions in Language Models

We use a scalable and unsupervised method called Sparse Autoencoders to find interpretable, monosemantic features in real LLMs (Pythia-70M/410M) for both residual stream and MLPs. We showcase monosemantic features, feature replacement for Indirect Object Identification (IOI), and use OpenAI's automatic interpretation protocol to demonstrate a significant improvement in interpretability.

Paper Overview

Sparse Autoencoders & Superposition

To reverse engineer a neural network, we'd like to first break it down into smaller units (features) that can be analysed in isolation. Using individual neurons as these units can be useful but neurons are often polysemantic, activating for several unrelated types of feature so just looking at neurons is insufficient. Also, for some types of network activations, like the residual stream of a transformer, there is little reason to expect features to align with the neuron basis so we don't even have a good place to start.

Overview of the methodology.

Toy Models of Superposition investigates why polysemanticity might arise and hypothesise that it may result from models learning more distinct features than there are dimensions in the layer, taking advantage of the fact that features are sparse, each one only being active a small proportion of the time. This suggests that we may be able to recover the network's features by finding a set of directions in activation space such that each activation vector can be reconstructed from a sparse linear combinations of these directions.

We attempt to reconstruct these hypothesised network features by training linear autoencoders on model activation vectors. We use a sparsity penalty on the embedding, and tied weights between the encoder and decoder, training the models on 10M to 50M activation vectors each. For more detail on the methods used, see the paper.

Automatic Interpretation

We use the same automatic interpretation technique that OpenAI used to interpret the neurons in GPT2 to analyse our features, as well as alternative methods of decomposition. This was demonstrated in a previous post but we now extend these results across the all 6 layers in Pythia-70M, showing a clear improvement over all baselines in all but the final layers. Case studies later in the paper suggest that the features ar...