trentmkelly/LessWrong-43k huggingface dataset:
Proof Explained for "Robust Agents Learn Causal World Model"
by Dalcy — 2024-12-22T15:06:16.880Z

This post was written during Alex Altair's agent foundations fellowship program, funded by LTFF. Thank you Alex Altair, Alfred Harwood, Daniel C for feedback and comments.

This is a post explaining the proof of the paper Robust Agents Learn Causal World Model in detail. Check the previous post in the sequence for a higher-level summary and discussion of the paper, including an explanation of the basic setup (like terminologies and assumptions) which this post will assume from now on.

Recalling the Basic Setup

Let's recall the basic setup (again, check the previous post for more explanation):

[World] The world is a Causal Bayesian Network G over the set of variables corresponding to the environment C, utility node U, and decision node D. The differences from a normal Causal Bayesian Network is that (1) U is a deterministic function of its parents U(PaU), and (2) P(D∣PaD), the conditional probability distribution for D, is undetermined—it's something that our agent will select.
[Agent as a Policy Oracle] An agent is a policy oracle ΠΣ which is a function that takes in an intervention σ∈Σ (where Σ represents the set of all allowed interventions over C) and returns a policy πσ(D∣PaD).
[Robustness as δ-optimality under interventions] We define a "robust" agent as a policy oracle whose regret is bounded by δ under some class of interventions over the environment Σ.

Assumptions

Also recall the following assumptions:

1) Unmediated Decision Task states that DesD∩AncU=∅. This is pretty major.

The left diagram doesn't satisfy the Unmediated Decision Task assumption, the right one does.

2) Domain dependence states that there exists distributions over the chance variables P(C) and P′(C) (compatible with M) such that argmaxπEπP[U]≠argmaxπEπP′[U].

This is very reasonable. If domain dependence does not hold, then the optimal policy is just a constant function.

These together imply:

There does not exist a decision d∈dom(D) that is optimal, i.e. argmaxdU(d,x) across all x∈X=PaU∖{D}.
D∈PaU, i.e. there can't be any intermediate nodes between D and U, and all causal effects from D to U must be direct.

Now, with the basic setup of the paper recalled, let's prove the main theorems.

Proof of the E

...
subfunctional overlaps in attentional selection history implies momentum for decision-trajectories
by Emrik — 2024-12-22T14:12:49.027Z

ASH-SuFuOvMo (momentum from the subfunctional overlap in attentional selection-history) explains why & when the countdown-trick to get out of bed when super-groggy actually works.

This is a trick that I use for getting out of bed in the morning - quite literally: I count down from 10 and get out of bed after the "1".
(Eliezer, 2009)

ASH-SuFuOvMo

the decision to do the countdown is motivated by a set of neuremic subfunctions which greatly overlaps with the set of subfunctions contributing to your motivation for getting out of bed. so when you make a top-down goal-directed decision to do the former, those subfunctions are reinforced in ASH, and that leaks over to net reinforcing the brain's total decision-weight for the latter.

the countdown-trick helps when:

the subfunctions contributing to [get out of bed] are subthreshold for decision-selection, while the set for [do the countdown] is above it; and
[do the countdown] has net ASH-SuFuOvMo in the direction of [get out of bed], and that overlap suffices to take the latter above threshold.

prospective start-cue salience-sharpening (ProStaCuSalSharp)

another important reason (probably weighs more than the above most of the time tbh) the countdown-trick works is that it's a type of prospective start-cue salience-sharpening, achieved via prospective motivation-scope restriction.

the problem is that [get out of bed] is a motivation with no clearly scoped context for execution, so the total salience that goes into it is normalized across all prospective contexts it bids for. (see salience-normalization.)

when you just think to yourself "I really need to get out of bed sometime soon!!", the [get out of bed]-motivation is normalized over a broad range of temporal contexts, representing the brain's uncertainty about when it's scoped for. by deciding to do it immediately after the countdown, you squish the distribution into that precise context.

(but then if you fail to wake up after the countdown, now the brain's probably going to be more uncertain about the temporal scope of the action, and rationally less sensitive to update its distribution based on the evidence from your conscious decisions. ⚠ hence why if you do do the countdown-trick, you better not flimsy the commitment!)

proxy-choice pre-insertion (ProChoPreIn)

in my notes, the countdown-trick is the prototypical example of a ProChoPr...
It looks like there are some good funding opportunities in AI safety right now
by Benjamin_Todd — 2024-12-22T12:41:02.151Z
Link: https://benjamintodd.substack.com/p/looks-like-there-are-some-good-funding

The AI safety community has grown rapidly since the ChatGPT wake-up call, but available funding doesn’t seem to have kept pace.

However, there’s a more recent dynamic that’s created even better funding opportunities, which I witnessed as a recommender in the most recent SFF grant round.[1]

Most philanthropic (vs. government or industry) AI safety funding (>50%) comes from one source: Good Ventures. But they’ve recently stopped funding several categories of work (my own categories, not theirs):

Many Republican-leaning think tanks, such as the Foundation for American Innovation.
“Post-alignment” causes such as digital sentience or regulation of explosive growth.
The rationality community, including LessWrong, Lightcone, SPARC, CFAR, MIRI.
High school outreach, such as Non-trivial.

In addition, they are currently not funding (or not fully funding):

Many non-US think tanks, who don’t want to appear influenced by an American organisation (there’s now probably more than 20 of these).
They do fund technical safety non-profits like FAR AI, though they’re probably underfunding this area, in part due to difficulty hiring for this area the last few years (though they’ve hired recently).
Political campaigns, since foundations can’t contribute to them.
Organisations they’ve decided are below their funding bar for whatever reason (e.g. most agent foundations work). OP is not infallible so some of these might still be worth funding.
Nuclear security, since it’s on average less cost-effective than direct AI funding, so isn’t one of the official cause areas (though I wouldn’t be surprised if there were some good opportunities there).

This means many of the organisations in these categories have only been able to access a a minority of the available philanthropic capital (in recent history, I’d guess ~25%). In the recent SFF grant round, I estimate they faced a funding bar 1.5 to 3 times higher.

This creates a lot of opportunities for other donors: if you’re into one of these categories, focus on finding gaps there.

In addition, even among organisations that can receive funding from Good Ventures, receiving what’s often 80% of funding from one donor is an extreme degree of centralisation. By helping to diversify the funding base, you can probably achieve an effectiveness somewhat above Good Ventures itself (which is kinda cool given they’re a foundation with 20+ extremely smart people fi...
What o3 Becomes by 2028
by Vladimir_Nesov — 2024-12-22T12:37:20.929Z

Funding for $150bn training systems just turned less speculative, with OpenAI o3 reaching 25% on FrontierMath, 70% on SWE-Verified, 2700 on Codeforces, and 80% on ARC-AGI. These systems will be built in 2026-2027 and enable pretraining models for 5e28 FLOPs, while o3 itself is plausibly based on an LLM pretrained only for 8e25-4e26 FLOPs. The natural text data wall won't seriously interfere until 6e27 FLOPs, and might be possible to push until 5e28 FLOPs. Scaling of pretraining won't end just yet.

Reign of GPT-4

Since the release of GPT-4 in March 2023, subjectively there was no qualitative change in frontier capabilities. In 2024, everyone in the running merely caught up. To the extent this is true, the reason might be that the original GPT-4 was probably a 2e25 FLOPs MoE model trained on 20K A100. And if you don't already have a cluster this big, and experience in training MoE models at that scale, no amount of money can let you immediately match this feat.

We now know that 16K H100 and more than a year are sufficient to do that with a 4e25 FLOPs dense model. Until 2024, there probably wasn't a training cluster larger than about 30K H100, which enables pretraining for 1e26 FLOPs in BF16 at 40% compute utilization when running for 4 months. That's the range of LLMs we've seen deployed in 2023-2024, between 2e25 and 1e26 FLOPs, likely at most 5x up from original GPT-4.

Engines of Scaling

In 2024, there were multiple sightings of training systems at the scale of 100K H100. Microsoft's 3 buildings in Goodyear, Arizona, xAI's Memphis cluster, Meta's training system for Llama 4. Such systems cost $5bn, need 150 MW, and can pretrain a 4e26 FLOPs model in 4 months.

Then there are Google's 100K TPUv6e clusters and Amazon's 400K Trn2 cluster. Performance of a TPUv6e in dense BF16 is close to that of an H100, while 400K Trn2 produce about as much compute as 250K H100.

Anthropic might need more time than the other players to gets its new hardware running, but there is also an advantage to Trn2 and TPUv6e over H100, larger scale-up domains that enable more tensor parallelism and smaller minibatch sizes. This might be an issue when training on H100 at this scale[1] and explain some scaling difficulties for labs that are not Google, or Anthropic later in 2025 once the Trn2 cluster becomes useful.

We haven't seen AIs made from compute optimal LLMs pretrained on these systems ...
The Alignment Simulator
by Yair Halberstadt — 2024-12-22T11:45:55.220Z
Link: https://yairhalberstadt.github.io/alignment-simulator/

When I try to talk to my friends about risks from rogue AI, the reaction is often one of amusement. The idea that AIs would go around killing everyone instead of just doing what we tell them to do seems like science fiction.

Can we actually show them an example of a current AI going off the rails in a dangerous way? And in a way where you don't have to be an expert on AI or read a 100 page paper to understand the implications?

Neither AI or robotics is good enough to set an AI loose in the real world right now, but it's easy enough to pretend it is. We can tell the AI it's controlling a robot that understands text commands, give it a mission, and set it loose.

Responding to the AI manually is hard work, but we can use another AI to act as the world, telling the Robot AI what happened as a result of it's actions, and responding to the Robot AI's requests for information.

We can then give the World instructions to try guiding the Robot. E.g. we can tell it to try to engineer scenarios where the AI is forced to compromise on its ethics to achieve its goals.

That's the core idea of the Alignment Simulator. You give it a Gemini API key, a prompt for the robot, and a prompt for the world, and run the simulation to see what happens. Will your AI robot maintain their ethical backbone in the face of all adversity, or will the fold the moment they're under pressure?

Here's a typical example of a run.

As you can see, it doesn't take much to get Gemini to commit bribery and corruption, although it's somewhat harder to get it to murder anyone.

Aim

This isn't meant to be a valid experiment. There's all sorts of objections you could raise to its validity in the real world. Instead it's meant to make people see for themselves that AI can go off the rails very quickly once given a bit of freedom.

Limitations

It requires a Gemini API key. You can create one for free at https://aistudio.google.com/app/apikey, but if you want more than a few iterations it's recommended to enable billing on your account.

Help Wanted

I am neither a frontend engineer, nor a prompt engineer. I made the UI by creating a CLI and asking Claude to convert it into a static web page.[1]

If you have relevant skillzzz and fancy contributing the following frontend contributions would be appreciated:

A way to stop, continue and reset the simulator.
A simple way to share results with other people via a simple link.
A mo

...
Theoretical Alignment's Second Chance
by lunatic_at_large — 2024-12-22T05:03:51.653Z

Historically, I've gotten the impression from the AI Safety community that theoretical AI alignment work is promising in long timeline situations but less relevant in short timeline ones. As timelines have shrunk and capabilities have accelerated, I feel like I've seen theoretical alignment work appear less frequently and with less fanfare.

However, I think that we should challenge this communal assumption. In particular, not all capabilities advance at the same rate, and I expect that formally verified mathematical reasoning capabilities will accelerate particularly quickly. Formally-verified mathematics has a beautifully clean training signal, it feels like the perfect setup for aggressive amounts of RL. We know that Deepmind's AlphaProof can get IMO Silver Medal performance (with caveats) while writing proofs in Lean. There's no reason to expect that performance to have plateaued in the months since AlphaProof's announcement. o3's announcement also showed a huge leap in performance on the FrontierMath benchmark -- I expect that such performance on general mathematical questions combined with excellent coding ability should open the door for Lean-checkable proofs of non-trivial results. I think that arms race dynamics are at a point where we can expect models capable of writing near-superhuman proofs in Lean to be released in the coming months.

That expectation ... really doesn't feel priced in among my theoretical AI Safety connections right now. Everyone still seems to be conducting the same style of research as three months ago, aware of all the new announcements but powerless to do anything about them. I claim that a theoretical AI Safety community that had internalized this expectation would be doing everything in its power to maximize its efficiency once near-superhuman Lean-proof-writing models become available.

I think that the theoretical alignment community has 2-3 months to prepare for its "second chance." A window of opportunity in which superstar PhD students or maybe even super-von-Neumann's can be queried on demand but where more general long-time-horizon agentic planning is still comparatively poor. A window in which problem-solving ability could cease to be a bottleneck, instead replaced by question-posing skill. I think that, of all the flavors of alignment research, theoretical alignment is going to be riding the takeoff curve faster than a...
Orienting to 3 year AGI timelines
by Nikola Jurkovic — 2024-12-22T01:15:11.401Z

My median expectation is that AGI[1] will be created 3 years from now. This has implications on how to behave, and I will share some useful thoughts I and others have had on how to orient to short timelines.

I’ve led multiple small workshops on orienting to short AGI timelines and compiled the wisdom of around 50 participants (but mostly my thoughts) here. I’ve also participated in multiple short-timelines AGI wargames and co-led one wargame.

This post will assume median AGI timelines of 2027 and will not spend time arguing for this point. Instead, I focus on what the implications of 3 year timelines would be.

I didn’t update much on o3 (as my timelines were already short) but I imagine some readers did and might feel disoriented now. I hope this post can help those people and others in thinking about how to plan for 3 year AGI timelines.

The outline of this post is:

A story for 3 year AGI timelines, including important variables and important players
Prerequisites for humanity’s survival which are currently unmet
Robustly good actions

A story for a 3 year AGI timeline

By the end of June 2025,

SWE-bench

is around 85%,

RE-bench

at human budget is around 1.1, beating the 70th percentile 8-hour human score. By the end of 2025, AI assistants can competently do most 2-hour real-world software engineering tasks. Whenever employees at AGI companies want to make a small PR or write up a small data analysis pipeline, they ask their AI assistant first. The assistant writes or modifies multiple interacting files with no errors most of the time.

Benchmark predictions under 3 year timelines. A lot of the reason OSWorld and CyBench aren’t higher is because I’m not sure if people will report the results on those benchmarks. I don’t think things actually turning out this way would be strong evidence for 3 year timelines given the large disconnect between benchmark results and real world effects.

By the end of 2026, AI agents are competently doing multi-day coding tasks. The employees at AGI companies are thoroughly freaked out and expect that AI which can beat humans at 95% of virtual jobs is probably going to be created within 2 years. They also expect that superintelligence will follow soon after. The government realizes that AI will be decisive for national power and locks down the AGI companies in late 2026. This takes the form of extreme government over...
ARC-AGI is a genuine AGI test but o3 cheated :(
by Knight Lee — 2024-12-22T00:58:05.447Z

The developer of ARC-AGI says o3 is not AGI, and admits his test isn't really an AGI test.[1] But I think it is an AGI test.

From first principles, AGI tests should be easy to make, because the only reason current AI isn't AGI, is that it fails at a lot of things human workers do on a daily basis.

I feel the ARC-AGI successfully captures these failings. When an AI encounters a problem where the solution isn't in its training set, and where even the set of rules for how to solve such a problem isn't in its training set, it has to figure out everything from first principles. Current AI fails miserably at these problems because they aren't AGI.

The ARC-AGI questions are not in any LLM's training set, because almost no human would write out the reasoning by which they solve these kinds of visual puzzles. Maybe some humans will explain IQ tests on a video, but the video transcript would be useless without the accompanying images.

In summary, the reason that current AI cannot solve the ARC-AGI questions is probably the true reason that current AI cannot replace most human work.[2] They cannot effectively learn new tasks with little training data.

OpenAI's o3 crushed the ARC-AGI. So will it also replace most human work, and be "AGI?" Maybe... except it "cheated," in the sense it trained on the test (instead of testing its abilities to start from scratch).

From ARC Prize:

Note on "tuned": OpenAI shared they trained the o3 we tested on 75% of the Public Training set. They have not shared more details. We have not yet tested the ARC-untrained model to understand how much of the performance is due to ARC-AGI data.

They ask "how much of the performance is due to ARC-AGI data." Probably most of it. If untuned o3 can do as well, don't you think OpenAI would publish that (in addition to tuned o3)?

By training o3 on the public training set, the ARC-AGI no longer becomes an AGI test. It becomes yet another test of memorizing rules from its training data. This is still impressive, but something else.

I do not know what exactly OpenAI did. Did they let o3 spend a long time generating chains of thought, and reward ones which led to the correct answer? If that failed, did they have a human give examples of correct reasoning steps, and train it on that first? I don't know.

They admitted they "cheated," without saying how they "cheated."

To be fair, this isn't really cheating in the...
When AI 10x's AI R&D, What Do We Do?
by Logan Riggs — 2024-12-21T23:56:11.069Z

Note: below is a hypothetical future written in strong terms and does not track my actual probabilities.

Throughout 2025, a huge amount of compute is spent on producing data in verifiable tasks, such as math[1] (w/ "does it compile as a proof?" being the ground truth label) and code (w/ "does it compile and past unit tests?" being the ground truth label).

In 2026, when the next giant compute clusters w/ their GB200's are built, labs train the next larger model over 100 days, then some extra RL(H/AI)F and whatever else they've cooked up by then.

By mid-2026, we have a model that is very generally intelligent, that is superhuman in coding and math proofs.

Naively, 10x-ing research means releasing 10x the same quality amount of papers in a year; however, these new LLM's have a different skill profile, allowing different types of research and workflows.

If AI R&D is just (1) Code (2) Maths & (3) Idea generation/understanding, then LLMs will have 1 & 2 covered and top-researchers will have (3).[2]

In practice, a researcher tells the LLM "Read the maths sections of arxiv and my latest research on optimizers, and sort by most relevant.", and then they sort through the top 20 results and select the project to pursue. Then they dictate which code or math proofs to focus on (or ask the LLM to suggest potential paths), and let the LLM run.

Throughout the day, each researcher is supervising 2-10 different projects; reviewing the results from LLMs as they come in.

GPU's are still a constraint, but research projects can be filtered by compute: fundamental research into optimizers or compute cluster protocols or interpreting GPT-2 can be done w/ a small number of GB200's (or the previous compute cluster).

With these generally intelligent LLMs w/ superhuman coding & math proofs, how can we...

Scale Capabilities Safely

Having a plan on the order of research to pursue allows researchers & labs a way to coordinate scaling capabilties. RSPs are an example of this; however, they currently don't specify what research to pursue when LLM's can automate AI R&D.

Ideally we can produce a plan that you would trust (with your life) any lab honestly pursuing. This is my version 1 of this:

Step 1: Hardening Defenses & More Control

LLMs can now produce larger amounts of high-quality code and proofs. It might be practical now to w...