trentmkelly/LessWrong-43k huggingface dataset:
Links and short notes, 2025-01-20
by jasoncrawford — 2025-01-21T16:10:51.813Z
Link: https://newsletter.rootsofprogress.org/p/links-and-short-notes-2025-01-20

Much of this content originated on social media. To follow news and announcements in a more timely fashion, follow me on Twitter, Threads, Bluesky, or Farcaster.

Contents

My writing (ICYMI)
Jobs and fellowships
Announcements
News
Events
Other opportunities
We are not close to providing for everyone’s “needs”
The printing press and the Internet
The ultimate form of travel
Five hot takes about progress
What could have been, for SF
Quick thoughts on AI
Links and bullets
Charts
Pics

My writing (ICYMI)

How sci-fi can have drama without dystopia or doomerism. “Concise but incredible resource” (@OlliPayne). “100 percent with Jason on this. If your sci-fi has technology as the problem it will put me to sleep” (@elidourado)

Jobs and fellowships

HumanProgress.org is hiring a research associate with Excel/Python/SQL skills “to manage and expand our database on human well-being” (@HumanProgress)
The 5050 program comes to the UK “to help great scientists and engineers become great founders and start deep tech startups,” in partnership with ARIA (@sethbannon)

Announcements

Core Memory, a new sci/tech media company from Ashlee Vance (@ashleevance)
“AI Summer”, a new podcast from Dean Ball (RPI fellow) and Timothy B. Lee (@deanwball)
Inference Magazine, a new publication on AI progress, with articles from writers including RPI fellow Duncan McClements (@inferencemag)

News

Matt Clifford has published an AI Opportunities Action Plan for the UK, and the PM has agreed to all its recommendations, including “AI Growth Zones” with faster planning permission and grid connections; accelerating SMRs to power AI infrastructure; procurement, visas & regulatory reform to boost UK AI startups; and removing barriers to scaling AI pilots in government (@matthewclifford)
The Manhattan Plan: NYC plans “to build 100,000 new homes in the next decade to reach a total of 1 MILLION homes in Manhattan” (@NYCMayor). “We’ve come a long way on housing,” says @danielgolliher

Events

Boom XB-1 first supersonic flight might be as soon as Jan 27, with livestream on X (@bscholl). After two supersonic tests, “we are all in on the big jet” (@bscholl)

Other opportunities

Dynomight is offering general mentoring (@dynomight7)

To read the rest, subscribe on Substack.
The Case Against AI Control Research
by johnswentworth — 2025-01-21T16:03:10.143Z

The AI Control Agenda,

in its own words

:

… we argue that AI labs should ensure that powerful AIs are controlled. That is, labs should make sure that the safety measures they apply to their powerful models prevent unacceptably bad outcomes, even if the AIs are misaligned and intentionally try to subvert those safety measures. We think no fundamental research breakthroughs are required for labs to implement safety measures that meet our standard for AI control for early transformatively useful AIs; we think that meeting our standard would substantially reduce the risks posed by intentional subversion.

There’s more than one definition of “AI control research”, but I’ll emphasize two features, which both match the summary above and (I think) are true of approximately-100% of control research in practice:

Control research exclusively cares about intentional deception/scheming; it does not aim to solve any other failure mode.
Control research exclusively cares about early transformatively-useful AI; it does not aim to generalize to any other AI. In particular, it does not aim to generalize to more powerful AI.

The core point of this post is: intentional scheming in early transformatively-useful AI is a relatively small slice of the AI X-risk probability mass, and therefore control research mostly doesn’t have much impact on X-risk.

The Model & The Problem

I’ll start with an intentionally somewhat-oversimplified model; the points made about this model do generalize to more realistic/complicated models (we’ll get to that later), but are much easier to explain in the simplified model.

In the simplified model, there are two strategically-important stages of AGI: early transformative AGI, and superintelligence.

Superintelligence is extremely dangerous by (strong) default. It will kill us or at least permanently disempower us, with high probability, unless we solve some technical alignment problems before building it. Humanity currently has no idea how to solve those technical problems, and they seem pretty hard.

Early transformative AI is much weaker than superintelligence, probably human-ish level. The big hope on which the control agenda is founded is that early transformative AI can be used to solve the hard technical problems of superintelligence. Insofar as that’s the hope, we don’t necessarily need to solve the hard technical problems...
Will AI Resilience protect Developing Nations?
by ejk64 — 2025-01-21T15:31:32.378Z

Position Piece: Most of the developing world lacks the institutional capacity to adapt to powerful, unsecure AI systems by 2030. Incautious model release could disproportionately affect these regions. Enhanced societal resilience in frontier AI states consequently provides no 'black cheque' for incautious release.

‘We should develop more societal resilience to AI-related harms’ is now a common refrain

in AI governance

. The argument goes that we should try to reduce the risks of large frontier models as far as possible, but some amount of harm is inevitable, and

technological developments

will always see potentially harmful models fall outside the regulatory net. The countermeasure, born both of practicality and necessity, is to develop

societal resilience through adaptation

to avoid, defend against and remedy harms from advanced AI systems.

This is a sensible idea, and one I think more people should do work on.

However, even if societal resilience against the harms of advanced AI systems (from here on, ‘societal resilience’) in frontier AI states was to improve significantly, it seems likely that the rest of the world is unlikely to develop at the same speed. Many societal resilience mechanisms depend on foundations which regions in the developing world do not have. If powerful models were used in these regions, they might still be used to perpetrate significant harms.

So if you only takeaway two things from this blog, it's:

Increased societal resilience in some areas does not 1) preclude the risks of AI models being misused in other areas and therefore 2) provide a blank cheque for those models to be deployed.
Regulation and deployment decisions based on the resilience of developed world states that ignored the vulnerability of nations that cannot implement similar protections might create severe harm.

This short exploratory post aims to do three things:

Examine some foundations of societal resilience and why they may be lacking in the developing world
Suggest some potential harms arising from policies assuming such resilience, and
Explore some avenues to mitigate these risks.

This is still a new research area, however, so I’m very keen to hear any feedback you might have.

‘Societal Resilience’ requires strong foundations

Successful implementation also depends on the existence of appropriate institutions for resolving c...
Sleep, Diet, Exercise and GLP-1 Drugs
by Zvi — 2025-01-21T12:20:06.018Z

As always, some people need practical advice, and we can’t agree on how any of this works and we are all different and our motivations are different, so figuring out the best things to do is difficult. Here are various hopefully useful notes.

Table of Contents

Effectiveness of GLP-1 Drugs.

What Passes for Skepticism on GLP-1s.

The Joy of Willpower.

Talking Supply.

Talking Price.

GLP-1 Inhibitors Help Solve All Your Problems.

Dieting the Hard Way.

Nutrients.

Are Vegetables a Scam?.

Government Food Labels Are Often Obvious Nonsense.

Sleep.

Find a Way to Enjoy Exercise.

A Note on Alcohol.

Focus Only On What Matters.

Effectiveness of GLP-1 Drugs

GLP-1 drugs are so effective that the American obesity rate is falling.

John Burn-Murdoch: While we can’t be certain that the new generation of drugs are behind this reversal, it is highly likely. For one, the decline in obesity is steepest among college graduates, the group using them at the highest rate.

In the college educated group the decline is about 20% already. This is huge.

What Passes for Skepticism on GLP-1s

This and our other observations are not easy to reconcile with this study, which I note for completeness and shows only 5% average weight loss in obese patients after one year. Which would be a spectacular result for any other drug. There’s a lot of data that says that in real world conditions you do a hell of a lot better on average than 5% here.

Here’s a strange framing from the AP: ‘As many as 1 in 5 people won’t lose weight with GLP-1 drugs, experts say.’

Jonel Aleccia: “I have been on Wegovy for a year and a half and have only lost 13 pounds,” said Griffin, who watches her diet, drinks plenty of water and exercises regularly. “I’ve done everything right with no success. It’s discouraging.”

Whether or not that is 13 more pounds than he would have lost otherwise, it’s not the worst outcome, as opposed to the 5 in 5 people who won’t lose weight without GLP-1 drugs. 4 out of 5 is pretty damn exciting. I love those odds.

Eliezer Yudkowsky offers caveats on GLP-1 drugs, saying by default you lose more lean muscle mass than under other weight loss (I presume you can fix this with diet and exercise, but if you could fully control those you wouldn’t need the drug), and it causes ‘fat cells to divide or proliferate’ so it could backfire. Still seems obviously worthwhile to me for...
We don't want to post again "This might be the last AI Safety Camp"
by Remmelt — 2025-01-21T12:03:33.171Z
Link: https://manifund.org/projects/11th-edition-of-ai-safety-camp

We still need more funding to be able to run another edition. Our fundraiser raised $6k as of now, and will end if it doesn't reach the $15k minimum, on February 1st. We need proactive donors.

If we don't get funded for this time, there is a good chance we will move on to different work in AI Safety and new commitments. This would make it much harder to reassemble the team to run future AISCs, even if the funding situation improves.
You can take a look at the track record section and see if it's worth it:

≥ $1.4 million granted to projects started at AI Safety Camp
≥ 43 jobs in AI Safety taken by alumni
≥ 10 organisations started by alumni

Edit to add: Linda just wrote a new post about AISC's theory of change.

You can donate through our Manifund page

You can also read more about our plans there.

If you prefer to donate anonymously, this is possible on Manifund.

Suggested budget for the next AISC

If you're a large donor (>$15k), we're open to let you choose what to fund.

Testimonials (screenshots from Manifund page)
On Responsibility
by silentbob — 2025-01-21T10:47:37.562Z

My view on the concept of responsibility has shifted a lot over the years. I’ve had three insights that brought me from my initial, very superficial and implicit understanding of responsibility, to the one I have today, which I consider more accurate, more practical, and more healthy.

Responsibility is Made Up

The first insight came while I was part of a dinner debate with a local philosopher and a few friends. We were discussing AI, of course, and the philosopher eventually brought up the question of responsibility: if a self-driving car malfunctions and ends up killing a pedestrian, who’s responsible? The human in the car? The car manufacturer? The programmers? The car itself? Can an AI ever be “responsible” for anything?

Considering this, I concluded that the question is a bit misguided. It sounds like a question about the world – as if the responsibility just lies somewhere and we can figure out where if we look hard enough. But at the end of the day, responsibility is just a tool that humans assign to others based on certain heuristics and shared understandings[1]. This doesn’t mean responsibility is not a practical tool. You can define it in certain ways and apply it in society, and if you do that in the right way then it ends up beneficial. But it was still a valuable insight for me that responsibility does not “exist” but is just a more-or-less-agreed-upon construct. And hence “Can an AI ever be responsible” is not so much a factual question. It’s not a question about the AI. It’s a question about when it becomes practical for humans to assign responsibility to an AI.

Responsibility as an Excuse

Let’s look at two scenarios that I encountered:

We were planning with around ten friends to meet the next day to get picked up by a bus. The bus would arrive at 10:05 the next morning, so Arnold suggested we meet at 10:00 at the bus stop. At which point Bianca said “Surely somebody will be too late then, let’s rather say we meet at 9:45 to be sure we all make it?”, to which Arnold responded “But then that’s the late person’s responsibility! Everybody should themselves be responsible to be on time”. I get where Arnold is coming from. But still, you’re deciding between two outcomes: We choose 10:00, which probably means somebody will miss the bus and hence miss out on the trip and everyone will be very sad about that and that pers

...
The ‘anti woke’ are positioned to win but can they capitalize?
by Hzn — 2025-01-21T09:52:50.673Z

All I want for Christmas is an impressive victory in the culture war w/o too much net social harm [N1]!

Epistemic status -- fact checking is for the weak. Ie I'm confident in what I'm saying but it hasn't been carefully vetted.

Rather than obsessing about AI doom let's think about some thing way more fun -- culture wars!

Let's pretend the goal is {an impressive victory in the culture war for the right side w/o too much net social harm}. Impressive victory for the right side is ex ante undefined; what's important is that it be ex post impressive. The question is how can such a thing actually be achieved.

Whether an impressive victory for the right side is preferable to an impressive victory for the left side or an intermediate outcome is beyond the scope of this post [N2].

In hindsight Donald Trump losing in 2020 was hugely advantageous to the right side of the culture wars. If the ‘anti woke’ are now able to capitalize, Biden's victory will likely go down as one of the Pyrrhic est of Pyrrhic victories. That Trump was able to make a comeback is triply beneficial to the ‘anti woke’ cause. 1) It's more impressive. 2) He came back with a more hardcore cabinet. 3) The left side is fatigued not energized & has lost the very strong support it previously had from institutions & the center.

But can the ‘anti woke’ capitalize? Irrespective of your personal stance, is there a way the right side can win w/o too much net social harm? (If you want to think of this as coherent extrapolated volition, I neither recommend that approach nor go I anti recommend it). So what could the right side do?

1) Get rid of (some) anti discrimination & hate crime laws. The Supreme Court maybe should declare most of these things unconstitutional. Anti discrimination laws in particular create a ratchet effect which is probably a key driver of wokeness. I think the SC may eventually do some thing like this just like the US eventually moved away from marijuana prohibition after like 80 years. Expected date of arrival -- 2044.

2) Defund academia. This could hurt me personally so I'm not completely unbiased. But academia is one area where the US is first class compared to other developed countries. If time lines were very long, I would support this given the possibility of pernicious influence over decades or even centuries. But long time lines seem unlikely barring partial societal collapse...
Almost all growth is exponential growth
by lemonhope — 2025-01-21T07:16:24.686Z

Why is almost everything either overwhelming or nonexistent? I can't step outside without stepping on a pigeon, but I haven't seen a cardinal in years. Topics are discussed either constantly or never. You bike 200 miles per week or never. You are probably cancer free or dead from cancer. Your old stocks and coins are either worthless or worth a fortune. Stuff from my old popular science magazines is either in the palm of my hand or nonexistent.

The answer is what you would expect. Memes and bike clubs and cancer and companies and technology and pigeons all tend to grow/improve/multiply at a rate proportional to their current size/capabilities/population. If you've forgotten precalc, that's the definition of exponential growth.

There is some resistance force against all things, but even an exponential resistance (eg by the immune system) rounds to zero eventually if it grows even slightly slower than its target. 1% growth per second is 10,000x more per day than 0.99% growth per second.

Of course earth can only nourish so many pigeons, but after the strong exponential is over, growth basically completely stops. There's not really a long linear phase or anything. Growth is just zero until it is exponential until it is zero again. This approximation is accurate enough to call it reality.

If someone offers you exponential returns on your investment then it's probably a scam, but banks do that and they just call it "compounding growth". That is a much more descriptive name.

It's a bit crazy to think AI capabilities will improve exponentially. I am a very reasonable person, so I just think they'll improve some amount proportional to their current level.
Arbitrage Drains Worse Markets to Feeds Better Ones
by Cedar — 2025-01-21T03:44:46.111Z

Also, why one account keeps running dry when you try to arbitrage two markets.

I was thinking about how inter-market arbitrage might affect one's account balances & the total amount of money on the markets arbitraged.

## The arbitrage.

Let's say I have accounts on prediction markets A and B, and I have discovered an arbitrage opportunity for an event that has a price/probability of a on A and b on B such that a>b.

This price difference allows us to arbitrage the markets by buying shares of NO on market A and buying shares of YES on market B. So currently our accounts would look like:

Cost:

Market A: (1-a) dollars spent

Market B: b dollars spent

Let the "true" probability of the event happening be `p`. Then, when the market resolves, we profit on:
- Market A: `(1-p)` dollars, in expectation
- Market B: `p` dollars, in expectation

So we can expect to make
- `(1-p) - (1-a) = a-p` dollars on market A
- `p-b` dollars on market B

The arbitrage works because `a-p + p-b = a-b` which is positive, and there is no risk because we always get `1` dollar back regardless of whether the event happens or not.

## The flow of money

If the two markets' probability estimates of the events average to the true probability of the event (such that `(a+b)/2 = p`), then we can expect our profit to be split evenly between market A and B, since if `a+b=2p`, then `a-p = p-b`.

If the markets are skewed such that A is more correct than B, then we can expect to make less money on A than B, as `a-p < p-b` when `2p < a+b`. [^1]

So in effect, by performing the arbitrage, we are draining money from the less accurate prediction market, and feeding into the more accurate one.

If we were to try to continue the arbitrage, we will have to withdraw money from B and feed it into A, **draining money from the less accurate market and putting money into the more accurate one.**

[^1]: Without loss of generality, the same applies if market B had a close-to-true probability estimate than A
On Contact, Part 1
by james.lucassen — 2025-01-21T03:10:54.429Z

Context: for fun (and profit?)

Basic Contact

Contact is a lightweight many-versus-one word guessing game. I was first introduced to it on a long bus ride several years ago, and since then it’s become one of my favorite games to play casually with friends. There are a few blog posts out there about contact, but I think it’s incredibly underrated.

The rules of contact are simple, but I often tell people it’s easier to learn by watching others play rather than by a verbal explanation of the rules. Nevertheless, here is a verbal explanation of the rules.

There is one player who is “the defender”. All of the other players are “attackers”, and work together to defeat the defender as quickly as possible. The defender cannot win, only stave off defeat.
The defender chooses a secret word at the start of the game. They reveal the first letter of the secret word to the attackers. The attackers win when they learn the defender’s secret word.
The attackers get information about the secret word by making “contact” with each other. To make contact, a pair of attackers must simultaneously count down from 3 and then say some word that starts with the prefix of the secret word that has been revealed so far. When attackers make contact, the defender must reveal the next letter of the secret word.
To prevent contact, the defender can try to guess the word that a pair of attackers are trying to make contact on before they say it. If the defender guesses the contact word, that word is blocked. Blocked words cannot be used to make contact.
To prevent blocking, attackers give each other vague clues about what word they have in mind. To indicate to an attacker who has given a clue that you think you know what word they have in mind and would like to make contact, say “contact!” and start the countdown.
The defender cannot block contact if the word the attackers are trying to make contact on is the secret word. If the attackers make contact on the secret word, they win.

Example Contact

We join our players in media res, at the start of the second round.

Defender: the word starts with H-E.
Attacker 1: double hockey sticks…
Attacker 2: contact!
Attackers 1 and 2 together: 3… 2…
Defender: hell!
Attackers 1: yeah that’s what I was thinking 🙁
Attacker 3: neutralized alpha particle?
Attacker 2: ????
Attacker 1: contact!
Attackers 1 and 3 together: 3… 2…
Defender: uh…
Attackers 1 and 3 together: 1… helium!
Defende

...