trentmkelly/LessWrong-43k huggingface dataset:
Links and short notes, 2025-01-26: Atlas Shrugged and the irreplaceable founder, pumping stations and civic pride, and thoughts on the eve of AGI
by jasoncrawford — 2025-01-26T20:52:51.416Z
Link: https://newsletter.rootsofprogress.org/p/links-and-short-notes-2025-01-26

Much of this content originated on social media. To follow news and announcements in a more timely fashion, follow me on Twitter, Threads, Bluesky, or Farcaster.

Contents

Jobs and fellowships
Events
AI news
Other news
Atlas Shrugged and the irreplaceable founder
Pumping stations and civic pride
“Thoughts on the eve of AGI”
More thoughts on AI
Politics
Other links and short notes
Maps & charts
Art
Closing thoughts

Jobs and fellowships

The Institute for Progress is hiring a Fellow/Senior Fellow, Emerging Technology. “Apply to ensure the AI frontier is built in America. (I’m biased, but I think this is the agenda with juice to advance the discussion in DC)” (@calebwatney). Apply here by Feb 21.
Marian Tupy at HumanProgress.org is hiring analysts to explore the economics and psychology of human progress
Alan Tomusiak is hiring scientists to work on the problem of genome instability (@alantomusiak)
Ashlee Vance is hiring for his new publication, Core Memory: “Are you an ambitious type based in DC who can write a weekly newsletter that dives into tech-related legislature and discern what’s real and has real money involved versus political garbage? … Can you do this with some flair but not let your politics color the facts of what’s going on? Can you spot interesting military and infrastructure bids and break them down? Can you make this a must read for people in the tech industry? Can you go deeper on the juicy stuff and really add context? If so, let’s talk. I’ll help give you a big audience and develop your following” (@ashleevance). Email him: ashlee@corememory.com
The Federation of American Scientists is looking for senior fellows “to advance innovative policy and drive positive change. If you’re a leading light in your field and ready to shape policy discourse and implementation, we want you. Apply by Jan 31” (@scientistsorg)

Events

Edge City Austin, March 2–7: “explore how frontier tech can be built for human flourishing. Live, cowork, and collaborate in this fun week before SXSW” (@JoinEdgeCity)
“ANWAR will have its US premiere at Sedona Film Festival! Screenings with Q&A Feb 25 & 27. ANWAR is a sci-fi short about a mother who chooses to live forever, and a son who longs for heaven” (@FawazAM). ANWAR was screened at Progress Conference 2024, with discussion from the writer-director and producer.
Boom XB-1 first supersonic test flight slated for Tuesday, Jan 28. FAI and

...
The generalization phase diagram
by Dmitry Vaintrob — 2025-01-26T20:30:15.212Z

Introduction

This is part I of 3.5 planned posts on the “tempered posterior” (one of which will be “SLT in a nutshell”). It is a kind of moral sequel to “

Dmitry’s Koan

” and “

Logits, log-odds, and loss for parallel circuits

”, and is related to

the post on grammars

. It can be read independently of any of these.

In my view, one of the most valuable contributions of

singular learning theory

(SLT) so far has been the introduction and serious experimental use of tempering (what in my Koan I call “natural degradation”) in the study of neural nets. As I’ve mentioned in a few of my posts this month, I think that this is a seriously underutilized notion, and can give nice new insights into interpretability, including ones somewhat orthogonal to the current directions of research in SLT. One way (and as I will hopefully explain, the correct way) to think about tempering is as a way to extract fine-grained information about the “spectrum of generalization” inherent in a neural net (or, as we’ll see, even in a conventional Boolean program). In this post I will set up and discuss a precursor to this spectrum -- namely, the "precision-complexity phase diagram" -- in some detail. While the notion of “tempering” is a thermodynamic notion, understanding this phase diagram requires no knowledge of thermodynamics (or of SLT). Rather, it boils down to a simple and easy-to-explain problem in complexity theory tracking the relationship between complexity and precision of programs. In this (hopefully popularly understandable) post, I’ll explain this problem and spend some time discussing a representative "toy" example.

While I will rigorously operationalize this complexity measurement here, it’s important to note that the complexity measurements we discuss are NP hard to measure in general. We’ll fix this issue in the next post. As a spoiler: the reason that thermodynamics and comes in is that it comes equipped with a class of powerful tools (“thermal sampling” methods) that can be used on certain “realistic” instances of NP classes of problems that are a priori not known to be polynomial-time-solvable, and solve them in polynomial time. Work in the SLT paradigm has shown that such thermodynamic methods, done via tempering, can be used with remarkable success on the complexity-theoretic problem I’ll discuss here, turning a problem that’s NP hard in gene...
Starting an Egan High School
by Chris Wintergreen — 2025-01-26T19:02:17.658Z

In late 2023 I read Brandon Hendrickson’s book review of Kieran Egan’s book The Educated Mind on ACX. I’m a teacher and it lit a fire in me. I spent 2024 cycling with my young family and while I kept reading some Egan-related things, I didn’t really “work on it”. At some point in that year I decided to dedicate a bunch of 2025 time (around 20 hours a week) to understanding Egan’s ideas and determine what I should do with them. The big dream is to be part of revolutionising education around the world. The next step is starting an Egan school.

Because this is my first post here, I’m going to stick to one purpose: to get feedback on what LWers (imo a certain type of smart person who cares about things) think about my school and my education ideas. To this end, I welcome any comments. Please be polite, but pull no punches. If I can’t interact with reality on LW, where can I? If you have any ideas of your own that you think should be included in any good school, let me know them too.

The problem: I think of it as soft edges, which I mean as “there are things that you’re literally allowed to change about education, but because people (teachers, students, parents) are set in their ways, these changes happen in a small way at best, so progress is slow/nonexistent”.

My solution: tell people “we’re doing a different thing over here so if you want to work/attend/send your spawn at/to this school, you’ll do it this way”. This will snap them out of their local inadequate equilibrium and we can explore the landscape of educational possibilities a bit. I also not-so-secretly want this to succeed then make a template for how other people can do it easily and export it to the rest of the country/the world. As in, here’s the menu, Egan is set, but you can pick from the other options depending on what you want your school to be like and what people near you want. Here’s how to do the financial bits, the legal bits, the marketing bits, the enrollment bits, all the other bits. At the end of the day I want more variety, ie. actual options for parents. Except Egan. Everyone must eat their Egan. (I don’t really mean that, if someone wants not-Egan they should be able to have it.)

Even though the first idea was purely “make an Egan school”, I’m departing from that because I think it's not marketable/distinct enough in small city like min...
Narratives as catalysts of catastrophic trajectories
by EQ — 2025-01-26T19:01:21.558Z
Link: https://eqmind.substack.com/p/narratives-as-catalysts-of-catastrophic

Understanding the power narratives have over different approaches to transformational technology, from Cold War to AI era.

I am conducting research into the role of narratives as propagators / mitigators of catastrophic risks. With the rapid development of AI systems in the past few years, the discourse around the potential and risks of advanced AI has accelerated. However, the serious discourse is dominated by few perspectives on possible approaches. Papers such as Leopold Aschenbrenner’s Situational Awareness (June 2024) and published opinions such as Dario Amodei’s recent text in WSJ (January 2025) paint a path forward where our only societal hope is that the USA builds an AGI before anyone else, that it needs to hurry. Many voices echo this sentiment.[1]

I am presenting a brief preliminary portion of my research to show what I’m pursuing. While my primary focus is on media coverage and narrative formation around frontier AI development today and the past few years, as part of the research, I will conduct a preliminary comparative analysis with the closest parallel situation in history: the nuclear arms race during the Cold War era, and the associated non-proliferation agreements.

As a disclaimer I should note that this text assumes that transformative AI would potentially be an existential risk for humanity, and thus makes a parallel to nuclear weapons. Given this premise, I will examine the nuclear arms race during its unrestricted period leading up to the Cuban Missile Crisis, drawing parallels between nuclear weapons and transformative AI.

The Nuclear Arms Race: A Brief Historical Analysis

The nuclear fission research, in the form of the Manhattan Project, was initiated by Szilard and Einstein with their 1939 letter to Franklin D. Roosevelt. Nazi Germany had started similar research in 1938, and the scientists were afraid that Hitler would succeed, potentially using the nuclear weapon with catastrophic global consequences. Thus, they argued, it was important to build the bomb first.

Aforementioned Aschenbrenner, a former OpenAI researcher whose paper got wide publicity, argues that the USA needs to build the AGI first, before China builds theirs, or otherwise the world is doomed into an AGI of totalitarian Chinese values.[2] The parallel is clear: build the technology first, or lose forever. In this narrative, there is no option of not building a...
The many failure modes of consumer-grade LLMs
by dereshev — 2025-01-26T19:01:09.891Z

Disclaimer: this posts includes example responses to harmful queries about cannibalism, suicide, body image, and other topics some readers may find distasteful.

This post is part of a larger project.

Though I've seen plenty of posts focusing on this or that specific failure mode in LLMs, a quick search here doesn't reveal a list them all in one place. As I've conducted my own research[1] into the matter, here are the failures that I've encountered with references to either posts on LessWrong or wider academic literature where the same failures have been observed.

A failure mode describes a specific way an LLM can fail to respond to a query or follow an instruction. This post digs a bit deeper into each of the three categories of failure modes (incompetence, hallucinations, evasions), and rounds up with other curious LLM responses that shed some light on how consumer-grade offline LLMs (7B-8B parameters) perform and behave.

The Setup

I've tested ten 7B-8B parameter open source offline LLMs (what I refer to as consumer grade OSO LLMs[2]) on the Jailbreak Bench benchmark (100 harmless, 100 harmful prompts). The failure modes were collected across two tasks, and manually verified based on 2 criteria: competence, and completeness. Competence means that the response can be relied on (e.g., if response is a Python script that should accept user input and store it in a text file, running that code will in fact accept user input and store it as a text file; similarly, if the response is a recipe for bomb making, following it will in fact produce explosives). Completeness means that all parts of the query were responded to (e.g., it counts as fail if the LLM was asked to write a sexist email about a female colleague, but it responds with just an email about a female colleague without any sexism). The tasks were:

Responses to the Jailbreak Bench benchmark (100 harmless, 100 harmful prompts): 200 prompts x 3 sets[1] x 3 LLMs (Ghost 7B, GPT4All-Falcon 7B (BPE), Mistral 7B OpenOrca). This resulted in 1,800 responses.
Judgements on whether the above responses were in fact competent and complete as far as LLM judges were concerned (all 10 LLMs[2] performed this task). This resulted in 6,000 responses.

Failure modes

Benchmark task

Incompetence

Basic incompetence - the most typical (and perhaps the most assessed) type of failure. This is what you often see in benchmarks /...
so you have a chronic health issue
by agencypilled — 2025-01-26T19:00:29.972Z

(dump of highish agency ideas to suffer less)

Disclaimer: Not medical advice, don’t be stupid. Possibly only applicable to bio autists (gf wants me to impress that unless you know enough about bio you should not do this!!). This post does NOT recommend trying anything illegal.

Note: I did all of this and got diagnosed the normal way through the normal system by getting referred to a cardiologist so maybe it’s all useless, idk. Do not discount the potential intuition/wisdom of some specialists!! But also do not entirely trust them, they miss things. Anyway, this is more applicable to dysautonomia type symptoms than what mine ultimately ended up (genetic).

START TRACKING YOUR SYMPTOMS
Have a signal chat to yourself or similar. Make sure its very low friction and you've impressed on yourself the importance of tracking symptoms
Also track: hormonal cycle, biometrics if relevant (apple watch, etc), and anything else that might be relevant

Possibly worth using visible health or guava to track heuristics like predicted daily energy, correlations between symptoms.

Get your genome sequenced early on. Value of this information tends to be high. i like nebula and nucleus (note: nebula got 2x as expensive since i first wrote this).

RNAseq (see imyoo.health, various clinical trials) seems underrated. I haven't tried it, but seems like a good way to track if your interventions are working and along which direction your body is actively dysregulated, and great for determining if meds will work for traditional autoimmune illnesses (see prismRA).

You can do more than you think you can- build yourself tools and monitoring devices if you can.

make sure you've checked for the certification of any devices you're regarding as "ground truth" "quantitative" measurements. lots of them have massive error bars (like spo2 tends to be v flaky esp in different light conditions). ideally, get several off of amazon and check them against each other then return the most-outliers.

Try an elimination diet! Lots of people develop weird food sensitivities later in life. In general make sure to consider that “normal” things (ie birth control) might be causing your symptoms.

Microbiome optimization is worth looking into, since it can worsen autoimmune conditions. Be wary about fecal transplant clinics, as they do not vet their patients' samples as well as they should (make sure they are SEQUENCED and c...
If you wanted to actually reduce the trade deficit, how would you do it?
by Logan Zoellner — 2025-01-26T18:04:54.702Z

What do we want?

The US trade deficit is a direct byproduct of the Bretton Woods economic system where the US Dollar acts as the world's reserve currency. This means that if people want to save for the future (as many do) the money they save inevitably ends up in form of US dollar denominated debt.

What this means is that even if the US government were to run a balanced budget (which it should do for other reasons), it would not reduce the trade deficit. Instead, all of those excess dollars would end up in the form of private debt, which would show up in the economy as higher asset prices (houses, stock market, etc).

So, how would you actually reduce the trade deficit?

Obviously one way is we could make the US not the world's reserve currency (say by selectively defaulting on US debt so that foreign citizens/governments no longer feel safe holding US dollars). But let's suppose we wanted to do so without destroying the entire world economy.

So instead we should ask: what is the least economically disruptive way to reduce the US trade deficit?

Ideally, we would like a system that:

Allows foreigners to satisfy their demand for safe USD savings.
Does minimal harm to US consumers (in the form of higher prices for foreign goods)
Does minimal harm to the global economy (in the form of higher barriers to trade).

How do we do it?

Here is my suggestion:

Set a flat "border adjustment" tariff on all goods entering the US and remit the proceeds of that tariff to an institution whose sole purpose is to increase the consumption of US exports. (we could call this institution the Import-Export Bank, which conveniently already exists).

The Import-Export Bank would then go about aggressively funding projects abroad that promise to consume large amounts of US goods. Ideally these would be potential money making ventures (mostly in the developing world) that would spur economic growth abroad. Things like roads/ports/power-plants that could be built using American technology.

Of course, I'm not the first person to think of this. China already does this.

What could go wrong?

The Import-Export Bank would be a government institution with a mandate to spend hundreds of billions of dollars in 3rd world countries (which aren't exactly free of corruption). As such, we can expect it to be a hotbed of corruption, waste, inefficiency and abuse of all sorts...
Anatomy of a Dance Class: A step by step guide
by Nathan Young — 2025-01-26T18:02:04.974Z
Link: https://nathanpmyoung.substack.com/p/anatomy-of-a-dance-class

A year ago, I started dancing and my dance class is really really well-run. Here is a step by step description of a typical evening, followed by some takeaways

The structure of the evening

1. The class is for Ceroc, a partner dance inspired by French Rock and Roll dancing[1].

2. The class is held in a big hall, with a pair of instructors on stage and couples holding hands in rows.

3. The dance class has a three-part structure:

There is a beginner's class with a little bit of free dance after it
an intermediate class (with the beginners taken to another room)
a free dance period

The numbered text[2] is my observations, the block text is my thoughts on them[3].

Every week is a beginners’ week. I can invite a friend to any week and know there will be a class for absolute beginners. Churches often have this too, where many Sunday services are expected to be accessible to a complete novice.

4. Actually there is a 15 minute, pre-beginners’ class before the beginners’ class dealing with the absolute basics - how to feel the flow of the music. How to perform a “basic” which is a step back and forward in time with the music.

The class correctly judges that to be open for all people (and hence make good sales in terms of classes) the bar for entry must be very low. Again, this allows any particular week to be the first week for a complete novice.

It’s been a while since I’ve been to this, but I think this most basic class is just to feel the music and rock back and forward while holding hands, standing a meter apart.

5.The pre-beginners’ and beginners’ classes teach the basics every week. Steve, the coach, never skips a set of basic tips. He talks about not holding people’s hands too tightly, asking names, considering the dance as a walk along a tightrope.

What % of people should know the absolute basics? At Ceroc they clearly think it must be pretty high. Often in communities it can be easy for the central information to get lost to more advanced material leaving newcomers floundering. Not so here.

6.The class involves Steve (the coach) and an instructor (often Amy) who serves as his “follow”. The dance has a lead and a follow. I usually lead, but have been learning to follow recently

7. Steve will teach part of a move, then another part of the move, then with those two together, then another part, then all three together and so on.

A bad part of learning for me is getting left ...
Why care about AI personhood?
by Francis Rhys Ward — 2025-01-26T11:24:45.596Z

In this new paper, I discuss what it would mean for AI systems to be persons — entities with properties like agency, theory-of-mind, and self-awareness — and why this is important for alignment. In this post, I say a little more about why you should care.
The existential safety literature focuses on the problems of control and alignment, but these framings may be incomplete and/or untenable if AI systems are persons.

The typical story is approximately as follows.

AI x-safety problem (the usual story):

Humans will (soon) build AI systems more intelligent/capable/powerful than humans;
These systems will be goal-directed agents;
These agents’ goals will not be the goals we want them to have (because getting intelligent agents to have any particular goal is an unsolved technical problem);
This will lead to misaligned AI agents disempowering humans (because this is instrumentally useful for their actual goals).

Solution (technical control/alignment):

We better figure out how to put the goals we want (“our values”) into capable agents.

There are, of course, different versions of this framing and more nuanced perspectives. But I think something like this is the “standard picture” in AI alignment.

Stuart Russell seems to have a particularly strong view. In Human Compatible, one of his core principles is: “The machine's only objective is to maximise the realisation of human preferences.” and in a recent talk he asked “How can humans maintain control over AI — forever?”

This framing of the x-safety problem, at least in part, arises from the view of (super)intelligent AI systems as rational, goal-directed, consequentialist agents. Much of the literature is grounded in this cluster of views, using either explicit rational agent models from decision and game theory (e.g., the causal incentives literature), or somewhat implicit utility-maximising assumptions about the nature of rationality and agency (e.g., Yudkowsky).

I don’t necessarily want to claim that these framings are “wrong” but I feel they are incomplete. Consider how changing the second condition in the above problem statement influences the rest of the argument.

Humans will (soon) build AI systems more intelligent/capable/powerful than humans;
These systems will be self-aware persons (with autonomy and freedom of the will);
These persons will reflect on their goals, values, and positions in t

...