trentmkelly/LessWrong-43k huggingface dataset:
The old memories tree
by Yair Halberstadt — 2025-03-05T19:03:59.498Z

This has nothing to do with usual Less Wrong interests, just my attempt to practice a certain style of creative writing I've never really tried before.

You're packing again. By now you have a drill. Useful? In a box. Clutter? In a garbage bag.

But there's some things that don't feel right in either. Under your bed, you find your old soft toy Fooby, now tattered, smelly, and stained. In your bedside table, there's a photo of you and your ex in Paris. Behind the dresser, an 18th birthday card from your nan. In the kitchen drawer, a key-ring your best friend bought for you when you were twelve.

You stare at them for a few minutes, then sigh and prepare to toss them in the garbage bag. Then you change your mind, dump them in a backpack with a coil of string, and head out on your bike.

You go down the road, around a corner, through an alleyway and along a dirt track for a couple of minutes. Ahead, you finally see the tree, a huge old thing spreading its canopy wide in an otherwise empty field.

Spring is newly come, and the fresh growth is mostly bare of memories. You quickly hang up the photo, keyring, and birthday card, but you feel that action isn't significant enough for Fooby.

Ducking, you enter the canopy and walk inwards. Past the fresh growth are last year's memories. Mostly photos, knickknacks, and old toys, but sometimes the artifacts speak of sadder stories...

A branch burdened with baby clothes, all still in their original packaging. A family photo with one member carefully blotted out. Even a funeral urn.

As you step further in, the toys start to be made of wood instead of plastic, and the clothes have rotted away.

At last, you reach the centre. Someone's hammered metal handholds into the trunk, and gingerly you start to climb, rising back out of the past towards the present. Here the artifacts get stranger. Broken musical instruments. A car key. An empty bottle of wine. A wedding ring.

About halfway up, you spy a 12-year-old girl sitting on a wide bough, cuddling a smelly rag-doll, her eyes red and wet. You scramble up beside her. Silently, you take the rag-doll and nestle it in a fork. Finally, you place Fooby in its lap.

You give the girl's hand a squeeze, and together you descend.
Reply to Vitalik on d/acc
by samuelshadrach — 2025-03-05T18:55:55.340Z
Link: http://samuelshadrach.com/?file=/raw/english/unimportant/reply_vitalik_dacc_reply.md

Update: This is a living document. Posted below is an older version. Link has the latest version.

2025-03-05

Vitalik recently wrote an article on his ideology of d/acc. This is impressively similar to my thinking so I figured it deserved a reply. (Not claiming my thinking is completely original btw, it has plenty of influences including Vitalik himself.)

Disclaimer
- This is a quickly written note. I might change my mind on this stuff tomorrow for all I know.

Two axes he identifies for differentially accelerating tech are:
- big group versus small group - prioritise accelerating tech that can be deployed by a small group rather than by a big group
- offense versus defense - prioritise accelerating tech that can be deployed for defence rather offense

I think I generally get where this is coming from and find these important ideas.

Some confusions from my side:

- Self-replication
- I am generally in favour of building self-sustaining social systems over not. Success of d/acc ultimately relies on followers of Vitalik's d/acc a) building only those tech that satisfy d/acc criteria and b) providing social approval to people who build tech as per d/acc criteria. For this system to be self-sustaining, point b) may need to be passed into the future long after all of d/acc's current followers (vitalik included) are dead. Self-replicating culture is possible to build but extremely difficult. Religions are among the oldest self-replicating cultures. Ideas such as markets and democracy have also successfully self-replicated for multiple centuries now. I'm unsure if this idea of d/acc being present in culture is alone sufficient to ensure people in year 2200 are still only building tech that satisfies d/acc criteria
- Often, culture is shaped by incentives IMO. If people of the future face incentives that make it difficult to follow d/acc, they might abandon it. It is hard for me to explain this idea in short, but it is something I consider very important. I would rather leave future generations with incentives to do a Thing, than just culture telling them to do a Thing.

- Terminal values
- To me the terminal values of all these galaxy-brain plans is likely preserving and growing timeless stuff like truth and empathy.
- Defensive tech provides truth a good defence as information is easy to replicate but hard to destroy. As long as multiple...
A Bear Case: My Predictions Regarding AI Progress
by Thane Ruthenis — 2025-03-05T16:41:37.639Z

This isn't really a "timeline", as such – I don't know the timings – but this is my current, fairly optimistic take on where we're heading.

I'm not fully committed to this model yet: I'm still on the lookout for more agents and inference-time scaling later this year. But Deep Research, Claude 3.7, Claude Code, Grok 3, and GPT-4.5 have turned out largely in line with these expectations[1], and this is my current baseline prediction.

The Current Paradigm: I'm Tucking In to Sleep

I expect that none of the currently known avenues of capability advancement are sufficient to get us to AGI[2].

I don't want to say the pretraining will "plateau", as such, I do expect continued progress. But the dimensions along which the progress happens are going to decouple from the intuitive "getting generally smarter" metric, and will face steep diminishing returns.

Grok 3 and GPT-4.5 seem to confirm this.

Grok 3's main claim to fame was "pretty good: it managed to dethrone Claude Sonnet 3.5.1 for some people!". That was damning with faint praise.
GPT-4.5 is subtly better than GPT-4, particularly at writing/EQ. That's likewise a faint-praise damnation: it's not much better. Indeed, it reportedly came out below expectations for OpenAI as well, and they certainly weren't in a rush to release it. (It was intended as a new flashy frontier model, not the delayed, half-embarrassed "here it is I guess, hope you'll find something you like here".)

GPT-5 will be even less of an improvement on GPT-4.5 than GPT-4.5 was on GPT-4. The pattern will continue for GPT-5.5 and GPT-6, the ~1000x and 10000x models they may train by 2029 (if they still have the money by then). Subtle quality-of-life improvements and meaningless benchmark jumps, but nothing paradigm-shifting.

(Not to be a scaling-law denier. I believe in them, I do! But they measure perplexity, not general intelligence/real-world usefulness, and Goodhart's Law is no-one's ally.)

OpenAI seem to expect this, what with them apparently planning to slap the "GPT-5" label on the Frankenstein's monster made out of their current offerings instead of on, well, 100x'd GPT-4. They know they can't cause another hype moment without this kind of trickery.

Test-time compute/RL on LLMs:

It will not meaningfully generalize beyond domains with easy verification. Some trickery like RLAIF and longer CoTs might provide some benefits, but they would be a fixed-size imp

...
On the Rationality of Deterring ASI
by Dan H — 2025-03-05T16:11:37.855Z
Link: nationalsecurity.ai/

I’m releasing a new paper “Superintelligence Strategy” alongside Eric Schmidt (formerly Google), and Alexandr Wang (Scale AI). Below is the executive summary, followed by additional commentary highlighting portions of the paper which might be relevant to this collection of readers.

Executive Summary

Rapid advances in AI are poised to reshape nearly every aspect of society. Governments see in these dual-use AI systems a means to military dominance, stoking a bitter race to maximize AI capabilities. Voluntary industry pauses or attempts to exclude government involvement cannot change this reality. These systems that can streamline research and bolster economic output can also be turned to destructive ends, enabling rogue actors to engineer bioweapons and hack critical infrastructure. “Superintelligent” AI surpassing humans in nearly every domain would amount to the most precarious technological development since the nuclear bomb. Given the stakes, superintelligence is inescapably a matter of national security, and an effective superintelligence strategy should draw from a long history of national security policy.

Deterrence

A race for AI-enabled dominance endangers all states. If, in a hurried bid for superiority, one state inadvertently loses control of its AI, it jeopardizes the security of all states. Alternatively, if the same state succeeds in producing and controlling a highly capable AI, it likewise poses a direct threat to the survival of its peers. In either event, states seeking to secure their own survival may preventively sabotage competing AI projects. A state could try to disrupt such an AI project with interventions ranging from covert operations that degrade training runs to physical damage that disables AI infrastructure. Thus, we are already approaching a dynamic similar to nuclear Mutual Assured Destruction (MAD), in which no power dares attempt an outright grab for strategic monopoly, as any such effort would invite a debilitating response. This strategic condition, which we refer to as Mutual Assured AI Malfunction (MAIM), represents a potentially stable deterrence regime, but maintaining it could require care. We outline measures to maintain the conditions for MAIM, including clearly communicated escalation ladders, placement of AI infrastructure far from population centers, transparency into datacenters, and more.

Nonproliferation

While deterr...
On OpenAI’s Safety and Alignment Philosophy
by Zvi — 2025-03-05T14:00:07.302Z

OpenAI’s recent transparency on safety and alignment strategies has been extremely helpful and refreshing.

Their Model Spec 2.0 laid out how they want their models to behave. I offered a detailed critique of it, with my biggest criticisms focused on long term implications. The level of detail and openness here was extremely helpful.

Now we have another document, How We Think About Safety and Alignment. Again, they have laid out their thinking crisply and in excellent detail.

I have strong disagreements with several key assumptions underlying their position.

Given those assumptions, they have produced a strong document – here I focus on my disagreements, so I want to be clear that mostly I think this document was very good.

This post examines their key implicit and explicit assumptions.

In particular, there are three core assumptions that I challenge:

AI Will Remain a ‘Mere Tool.’

AI Will Not Disrupt ‘Economic Normal.’

AI Progress Will Not Involve Phase Changes.

The first two are implicit. The third is explicit.

OpenAI recognizes the questions and problems, but we have different answers. Those answers come with very different implications:

OpenAI thinks AI can remain a ‘Mere Tool’ despite very strong capabilities if we make that a design goal. I do think this is possible in theory, but that there are extreme competitive pressures against this that make that almost impossible, short of actions no one involved is going to like. Maintaining human control is to try and engineer what is in important ways an ‘unnatural’ result.

OpenAI expects massive economic disruptions, ‘more change than we’ve seen since the 1500s,’ but that still mostly assumes what I call ‘economic normal,’ where humans remain economic agents, private property and basic rights are largely preserved, and easy availability of oxygen, water, sunlight and similar resources continues. I think this is not a good assumption.

OpenAI is expecting what is for practical purposes continuous progress without major sudden phase changes. I believe their assumptions on this are far too strong, and that there have already been a number of discontinuous points with phase changes, and we will have more coming, and also that with sufficient capabilities many current trends in AI behaviors would reverse, perhaps gradually but also perhaps suddenly.

I’ll then cover their five (very good) core principles.

I call ...
Contra Dance Pay and Inflation
by jefftk — 2025-03-05T02:40:03.100Z

Max Newman is a great contra
dance musician, probably best known for playing guitar in the

Stringrays, who recently wrote a

piece
on dance performer pay, partly prompted by my

post last
week. I'd recommend reading it and the comments for a bunch of
interesting discussion of the tradeoffs involved in pay.

One part that jumped out at me, though, is his third point:

3) Real World Compensation is Behind

Risking some generalizing and over-simplifying, any dance performer
could tell you that over the past 10 (20!) years, the compensation
numbers have been sticky, sometimes static. In real terms,
compensation on the whole has not kept up with inflation.

This is quite important: if pay is decreasing in real terms then it's
likely that the dance community is partly coasting off of past
investment in talent and we shouldn't expect that to continue. Except
when I look back over my own compensation, however, I don't see a
decrease. For dance weekends, counting only weekends that included
travel, my averages have been (in constant January 2025 dollars):

Year
Mean
Count

2014
$600
2

2015
$732
5

2016
$804
5

2017
$879
5

2018
$798
3

2019
$833
3

2022
$831
2

2023
$789
5

2024
$893
3

I wouldn't put too much weight on the low numbers for 2014 and 2015:
initially the Free Raisins
weren't too sure what the going rates were and probably ended up a bit
on the low side.

What about dances that aren't special events? My record keeping for
evening dances isn't quite good enough to make these numbers easy to
pull, but I do have good data for tours:

Date
Mean
Count

2012-08
$165
7

2013-07
$155
7

2014-07
$189
11

2019-05
$134
6

2024-02
$192
6

2024-07
$115
7

2025-02
$226
4

One thing to keep in mind is that tour payments depend on where in the
country you're touring, and are correlated. For example the 2024-07 tour was
through a lower cost of living area (Rochester, Pittsburgh,
Bloomington, St Louis, Cincinnati, Indianapolis) while the 2025-02
tour was the opposite (Baltimore, DC, Bethlehem, NYC). But here
as well I don't see payment failing to keep up with inflation.

One more place I can look for data is what BIDA has been paying. The structure is a guarantee
(the minimum performers are paid, regardless of what attendees do) and
then a potential bonus (originally ...
*NYT Op-Ed* The Government Knows A.G.I. Is Coming
by worse — 2025-03-05T01:53:00.165Z
Link: https://www.nytimes.com/2025/03/04/opinion/ezra-klein-podcast-ben-buchanan.html

All around excellent back and forth, I thought, and a good look back at what the Biden admin was thinking about the future of AI.

an excerpt:

[Ben Buchanan, Biden AI adviser:] What we’re saying is: We were building a foundation for something that was coming that was not going to arrive during our time in office and that the next team would have to, as a matter of American national security — and, in this case, American economic strength and prosperity — address.

[Ezra Klein, NYT:] This gets to something I find frustrating in the policy conversation about A.I.

You start the conversation about how the most transformative technology — perhaps in human history — is landing in a two- to three-year time frame. And you say: Wow, that seems like a really big deal. What should we do?

That’s when things get a little hazy. Maybe we just don’t know. But what I’ve heard you kind of say a bunch of times is: Look, we have done very little to hold this technology back. Everything is voluntary. The only thing we asked was a sharing of safety data.

Now in come the accelerationists. Marc Andreessen has criticized you guys extremely straightforwardly.

Is this policy debate about anything? Is it just the sentiment of the rhetoric? If it’s so [expletive] big, but nobody can quite explain what it is we need to do or talk about — except for maybe export chip controls — are we just not thinking creatively enough? Is it just not time? Match the calm, measured tone of this conversation with our starting point.

I think there should be an intellectual humility here. Before you take a policy action, you have to have some understanding of what it is you’re doing and why.

So it is entirely intellectually consistent to look at a transformative technology, draw the lines on the graph and say that this is coming pretty soon, without having the 14-point plan of what we need to do in 2027 or 2028.

Chip controls are unique in that this is a robustly good thing that we could do early to buy the space I talked about before. But I also think that we tried to build institutions, like the A.I. Safety Institute, that would set the new team up, whether it was us or someone else, for success in managing the technology.

Now that it’s them, they will have to decide as the technology comes on board how we want to calibrate this under regulation.

What kinds of decisions do you think they will have to make in the

...
Could this be an unusually good time to Earn To Give?
by TomGardiner — 2025-03-04T21:51:19.148Z
Link: https://forum.effectivealtruism.org/posts/dnKvAJcRko9ZPJcA6/could-this-be-an-unusually-good-time-to-earn-to-give

I think there could be compelling reasons to prioritise Earning To Give highly, depending on one's options. This is a "hot takes" explanation of this claim with a request for input from the community. This may not be a claim that I would stand by upon reflection.

I base the argument below on a few key assumptions, listed below. Each of these could be debated in their own right but I would prefer to keep any discussion of them outside this post and its comments. This is for brevity and because my reason for making them is largely a deferral to people better informed on the subject than I. The Intelligence Curse by Luke Drago is a good backdrop for this.

Whether or not we see AGI or Superintelligence, AI will have significantly reduced the availability of white-collar jobs by 2030, and will only continue to reduce this availability.
AI will eventually drive an enormous increase in world GDP.
The combination of these will produce a severity of wealth inequality that is both unprecedented and near-totally locked-in.

If AI advances cause white-collar human workers to become redundant by outperforming them at lower cost, we are living in a dwindling window in which one can determine their financial destiny. Government action and philanthropy notwithstanding, one's assets may not grow appreciably again once their labour has become replaceable. An even shorter window may be available for starting new professions as entry-level jobs are likely the easiest to automate and companies will find it easier to stop hiring people than start firing them.

That this may be the fate of much of humanity in the not-too-distant future seems really bleak. While my ear is not the closest to the ground on all things AI, my intuition is that humanity will not have the collective wisdom to restructure society in time to prevent this leading to a technocratic feudal hierarchy. Frankly, I'm alarmed that having engaged with EA consistently for 7+ years I've only heard discussion of this very recently. Furthermore, the Trump Administration has proven itself willing to use America's economic and military superiority to pressure other states into arguably exploitative deals (tariffs, offering Ukraine security guarantees in exchange for mineral resources) and shed altruistic commitments (foreign aid). My assumption is that if this Administration, or a similar successor, oversaw the unvei...
What is the best / most proper definition of "Feeling the AGI" there is? 
by Annapurna — 2025-03-04T20:13:40.946Z

I really like this phrase. I feel very identified with it. I have used it at times to describe friends who have that realization of where we are heading. However when I get asked what Feeling the AGI means, I struggle to come up with a concise way to define the phrase.

What are the best definitions you have heard, read, or even come up with to define Feeling the AGI?
Energy Markets Temporal Arbitrage with Batteries
by NickyP — 2025-03-04T17:37:56.804Z

Epistemic Status: I am not an energy expert, and this was done rather briefly. All analysis uses pricing data specific to Ireland, but some general ideas are likely applicable more broadly. Data is true as of March 2025. Where there are uncertainties I try to state them, but there are likely some factual errors.

TL;DR:

Now that batteries have gotten so cheap (approx 50-100€/kWh), it seems surprising that there is so much temporal price discrepancy to buy energy at low prices and sell/use energy at higher prices. In Ireland, the arbitrage opportunity per year seems to be on the order of ~45€/kWh per year on day-ahead wholesale markets, possibly more on balancing markets, and ~62€/kWh for households.

The main factors seem to be:

Costs for connecting batteries (hybrid inverters, battery management systems, etc..) seem to add up to more than the cost of the batteries now in most cases
It takes a while for the market to update.

For the wholesale market: Installing GWh scale battery storage locations takes a lot of time to provision
For households: pre-packaged solutions seem to be much more expensive than what one could do DIY, leading to much slower payback times

While the naive ROI seems to be within 2 years, equipment and other costs increase the payback period to more like 3-7 years. Overall, it seems like while there is some investigation and investment in this space, it does not seem to be matching the potential demand, with a possible main factor that may lower longer-term profitability being that there will be an interlink built between Ireland and France in 2026, which will likely reduce the price volatility somewhat.

Battery Cell Costs

Battery manufacturing capacity has gone up a lot, and prices have come down. In 2010, prices were around 1191$/kWh, dropping to 384 $/kWh in 2015, and 137 $/kWh in 2020 (accoriding to Statista). However, prices have continued to drop, and priced can be as low as $53 per kilowatt-hour (Fanatical Futurist). China's overall battery manufacturing capacity has exploded and is reported to exceed demand by more than double.

As a starting point, we need some examples of how much batteries actually cost. I try find some examples online. I do not endorse any of these sellers, and have never bought batteries from any of them, but I find typical discussion using "online approximate figures" as not being sufficiently concrete.

Anyway, these are s...