trentmkelly/LessWrong-43k huggingface dataset:
"Self-Blackmail" and Alternatives
by jessicata — 2025-02-09T23:20:19.895Z
Link: https://unstableontology.com/2025/02/09/self-blackmail-and-alternatives/

Ziz has been in the news lately. Instead of discussing that, I'll discuss an early blog post, "Self-Blackmail". This is a topic I also talked with Ziz about in person, although not a lot.

Let's start with a very normal thing people do: make New Year's resolutions. They might resolve that, for example, they will do strenuous exercise at least 2 times a week for the next year. Conventional wisdom is that these are not very effective.

Part of the problem is that breaking commitments even once cheapens the commitment: once you have "cheated" once, there's less of a barrier to cheating in the future. So being sparing about these explicit commitments can make them more effective:

I once had a file I could write commitments in. If I ever failed to carry one out, I knew I'd forever lose the power of the file. It was a self-fulfilling prophecy. Since any successful use of the file after failing would be proof that a single failure didn't have the intended effect, so there'd be no extra incentive.

If you always fulfill the commitments, there is an extra incentive to fulfill additional commitments, namely, it can preserve the self-fulfilling prophecy that you always fulfill commitments. Here's an example in my life: sometimes, when I have used addictive substances (e.g. nicotine), I have made a habit of tracking usage. I'm not trying to commit not to use them, rather, I'm trying to commit to track usage. This doesn't feel hard to maintain, and it has benefits, such as noticing changes in the amount of substance consumed. And it's in an area, addictive substances, where conventional wisdom is that human intuition is faulty and willpower is especially useful.

Ziz describes using this technique more extensively, in order to do more work:

I used it to make myself do more work. It split me into a commander who made the hard decisions beforehand, and commanded who did the suffering but had the comfort of knowing that if I just did the assigned work, the benevolent plans of a higher authority would unfold. As the commanded, responsibility to choose wisely was lifted from my shoulders. I could be a relatively shortsighted animal and things'd work out fine. It lasted about half a year until I put too much on it with too tight a deadline. Then I was cursed to be making hard decisions all the time. This seems to have improved my decisions, ultimately.

Compared to my "satisf...
Altman blog on post-AGI world
by Julian Bradshaw — 2025-02-09T21:52:30.631Z
Link: https://blog.samaltman.com/three-observations

First part just talks about scaling laws, nothing really new. Second part is apparently his latest thoughts on a post-AGI world. Key part:

While we never want to be reckless and there will likely be some major decisions and limitations related to AGI safety that will be unpopular, directionally, as we get closer to achieving AGI, we believe that trending more towards individual empowerment is important; the other likely path we can see is AI being used by authoritarian governments to control their population through mass surveillance and loss of autonomy.
Ensuring that the benefits of AGI are broadly distributed is critical. The historical impact of technological progress suggests that most of the metrics we care about (health outcomes, economic prosperity, etc.) get better on average and over the long-term, but increasing equality does not seem technologically determined and getting this right may require new ideas.

In particular, it does seem like the balance of power between capital and labor could easily get messed up, and this may require early intervention. We are open to strange-sounding ideas like giving some “compute budget” to enable everyone on Earth to use a lot of AI, but we can also see a lot of ways where just relentlessly driving the cost of intelligence as low as possible has the desired effect.

Anyone in 2035 should be able to marshall the intellectual capacity equivalent to everyone in 2025; everyone should have access to unlimited genius to direct however they can imagine.

Edit to add commentary:
That last part sounds like he thinks everyone should be on speaking terms with an ASI by 2035? If you just assume alignment succeeds, I think this is a directionally reasonable goal - no permanent authoritarian rule, ASI helps you as little or as much as you desire.
Forecasting newsletter #2/2025: Forecasting meetup network
by NunoSempere — 2025-02-09T18:07:51.514Z
Link: https://forecasting.substack.com/p/forecasting-newsletter-22025-forecasting

Highlights

Forecasting meetup network

(

a

) looking for volunteers. If you want to host a meetup in your city, send an email at forecastingmeetupnetwork@gmail.com.
Caroline Pham moves up to

Chairman of the CFTC

. She is much friendlier to prediction markets and has spent years writting dissents againsts regulatory overreach.
“

Yunaplan for liquidity

” makes subtle but very neat mechanism change for Manifold cash markets.

Prediction markets and forecasting platforms

The Yunaplan for Liquidity

(

a

) is a proposal on Manifold to subtly replace the default of (moving probabilities in a market maker or order book) with (placing a short-lived limit order which bots can then fill). I’m not sure to what extent this will work, but this is a subtle but very clever UI and mechanistic improvement, particularly since Manifold has cultivated a good bots ecosystem.

The State of Metaculus

(

a

) is a pretty high signal edition of their newsletter. They have gotten many more users with recent tournaments, were mentioned by the US CDC, benchmarked AI forecasters against users, hosted workshops. And my friend Molly has an

AI readiness index

(

a

), potentially one of many to come.

Kalshi has initially got a temporary advantage over competitors by setting regulators on competitors, but that initial advantage was easy to replicate by Interactive Brokers, Crypto.com, etc. Now they have added

Donald Trump Jr.

to their board of advisors, which is more difficult to copy.

Kalshi gained some market position via offering sports betting on

Robinhood

(

a

), but these were just

halted

. Kalshi also has prediction markets on the

2028 republican

(

a

) and

democratic

(

a

) candidates: these are possible because Kalshi is offering interest on positions.

The folks at the

American Civics Exchange

(

a

) would like me to remind readers of their existence. They are open to US traders with a bankroll of >= $10M, or >= $1M for “hedging purposes”. If you sign up

here

(

a

) I may get a small bonus. It could be worth signing up in advance in case one wants to actually trade when things are happening, rather than having to sign up then. But on the other hand, if you are a millionnaire your time is pretty valuable, so not sure that completing sign up flows is a good use of it.

Polymarket is

looking for writers

(

a

) for their publication The Oracle, with 55K subscribers.

New platforms

Glimt

(

a

) is a forecasting platform for predicting ...
How identical twin sisters feel about nieces vs their own daughters
by Dave Lindbergh — 2025-02-09T17:36:25.830Z

(cross posted from https://mugwumpery.com/how-identical-twin-sisters-feel-about-nieces-vs-their-own-daughters/)

It seems to be generally assumed that twin sisters feel the same way as other sisters – closer to their own children.

But per Hamilton/Trivers, they shouldn’t. They should feel equally related and care equally about daughters and nieces.

Identical twins share 100% of their genes, and their nieces are just as closely related as their own daughters (r=0.5 for both, vs. r=0.25 for nieces of non-identical sisters).

I’m told that in fact twin sisters usually care more about their own children than about their nieces.

I suspect that this is because identical twins are rare and our genes just program us to assume nieces are less closely related than our children, even tho with identical twins it’s not true.
Two hemispheres - I do not think it means what you think it means
by Viliam — 2025-02-09T15:33:53.391Z

I am going to address some misconceptions about brain hemispheres -- in popular culture, and in Zizian theory. The latter, because the madness must stop. The former, because it provided a foundation for the latter.

*

Two hemispheres in popular culture

About 99% of animals are bilaterally symmetric -- the left side and the right side of the body are approximately each other's mirror images. The symmetry is not perfect. For example the human heart is situated slightly on the left side, and its left and right halves have slightly different functions. But in general, it seems like once Evolution Fairy has decided that bilateral symmetry is a good idea, it was easier to keep designing all new organs symmetrically. Butterflies have pairs of wings, humans have a pair of hands.

Sometimes it means that organs come in pairs: we have a pair of eyes, and several pairs of ribs. Sometimes it means that organs have two connected parts, such as the left lung and the right lung. Some organs along the sagittal plane only have one approximately symmetric part, for example the stomach or the vertebrae.

The brain is not an exception to this rule. The parts that come in pairs include the cerebral hemispheres, the cerebellar hemispheres, basal ganglia, thalami, hippocampi, amygdalae, etc. The symmetric parts on the sagittal plane include corpus callosum, medulla, pons, midbrain, pineal gland, hypothalamus, hypophysis, etc.

Popular culture simplifies this as "we have two brains".

Simplifications are inevitable, but I wish humanity had settled on something less confusing and woo-inspiring, such as "the brain has two sides". Instead, it is known that:

the left brain is logical, the right brain is creative
the left brain is mathematical, the right brain is artistic
the left brain understands languages, the right brain understands music
the left brain prefers Android, the right brain prefers iPhone

...okay, I admit that I made up the last one. The rest is an oversimplification.

To explain, let's consider the language. Broca's area (speech and grammatical structure) and Wernicke’s area (comprehension of written and spoken language) are located in one cerebral hemisphere (almost always both in the same one), usually in the left hemisphere (for over 95% of right-handers and about 70% of left-handers). The left angular gyrus is essential for reading, writing, and connecting written symbols with their m...
The Structure of Professional Revolutions
by SebastianG  — 2025-02-09T13:23:01.059Z

An expert is not merely someone who has memorized data but someone who has internalized the structure of knowledge itself. This is why we call them PhDs—Doctors of Philosophy. Their expertise extends beyond isolated facts to the organizing principles that connect those facts, allowing them to wield knowledge in novel ways. While this definition is not airtight, it captures one key aspect of the priesthood that interests me: the structure of knowledge transmission.

The Priesthood of Knowledge

The concept of a priesthood in professional spheres comes from two sources: Astral Codex Ten’s recent discussion on expertise as a social function and Richard Posner’s 1988 article on Professionalisms.

Posner argues that professions—law, medicine, academia—function not merely as knowledge repositories but as cartels that control access to specialized knowledge, determining who is permitted to practice and what constitutes legitimate expertise. This gatekeeping is justified on the basis of quality control but often functions as self-preservation. Professionals claim authority by mastering a body of knowledge and its internal logic, which they enforce through credentials, ethical codes, and disciplinary norms.

Astral Codex Ten extends this idea, describing expertise as a priesthood—not in a pejorative sense, but as a structural necessity. Like religious orders, professions sustain their authority by maintaining interpretive dominance over their domain. They decide what counts as valid knowledge, who is qualified to interpret it, and how it should be disseminated. This ensures stability and continuity but also creates rigidity: professionals, like priests, can become blind to shifts in the structure of knowledge itself.

This priesthood model helps explain why professional authority is periodically challenged—when the structure of knowledge transmission shifts faster than the established class can adapt.

The Problem of Deciderization

This brings us to the problem of deciderization, a term borrowed from David Foster Wallace’s introduction to Best American Essays 2007. DFW describes “deciderization” as the process by which a system determines which stories, ideas, or facts matter and which do not—what is deemed important enough to be recorded, discussed, and remembered.

Deciderization is the hidden function of every knowledge system. It determines what makes it into textbooks, what ge...
Gary Marcus now saying AI can't do things it can already do
by Benjamin_Todd — 2025-02-09T12:24:11.954Z
Link: https://benjamintodd.substack.com/p/gary-marcus-says-ai-cant-do-things

January 2020, Gary Marcus wrote GPT-2 And The Nature Of Intelligence, demonstrating a bunch of easy problems that GPT-2 couldn’t get right.

He concluded these were “a clear sign that it is time to consider investing in different approaches.”

Two years later, GPT-3 could get most of these right.

Marcus wrote a new list of 15 problems GPT-3 couldn’t solve, concluding “more data makes for a better, more fluent approximation to language; it does not make for trustworthy intelligence.”

A year later, GPT-4 could get most of these right.

Now he’s gone one step further, and criticised limitations that have already been overcome.

Last week Marcus put a series of questions into chatGPT, found mistakes, and concluded AGI is an example of “the madness of crowds”.

However, Marcus used the free version, which only includes GPT-4o. That was released in May 2024, an eternity behind the frontier in AI.

More importantly, it’s not a reasoning model, which is where most of the recent progress has been.

For the huge cost of $20 a month, I have access to GPT-o1 (not the most advanced model OpenAI offers, let alone the best that exists).

I asked GPT-o1 the same questions Marcus did and it didn’t make any of the mistakes he spotted.

First he asked it:

Make a table of every state in the US, including population, area and median household income, sorted in order of median household income.

GPT-4o misses out a bunch of states. GPT-o1 lists all 50 (full transcript).

Then he asked for a column added on population density. This also seemed to work fine.

He then made a list of Canadian provinces and asked for a column listing how many vowels were in each name.

I was running out of patience, so asked the same question about the US states. This also worked:

To be clear, there are probably still some mistakes in the data (just as I’d expect from most human assistants). The point is that the errors Marcus identified aren’t showing up.

He goes on to correctly point out that agents aren’t yet working well. (If they were, things would already be nuts.)

And list some other questions o1 can already handle.

Reasoning models are much better at these kinds of tasks, because they can double check their work.

However, they’re still fundamentally based on LLMs – just with a bunch of extra reinforcement learning.

Marcus’ Twitter bio is “Warned everyone in 2022 that scaling would run out.” I agree scaling will run out at some ...
How do you make a 250x better vaccine at 1/10 the cost? Develop it in India.
by Abhishaike Mahajan — 2025-02-09T03:53:17.050Z
Link: https://www.owlposting.com/p/how-do-you-make-a-250x-better-vaccine

(I made a vaccinology/policy-based podcast! A very long one! If you'd like to avoid the summary below, here is the

Youtube link

and

Substack link

.).

Summary: There's a lot of discussion these days on how China's biotech market is on track to bypass the US's. I wondered: shouldn't we have observed the exact same phenomenon with India? It has seemingly all the same ingredients: low cost of labor, smart people, and a massive internal market.

Yet, the Indian biotech research scene is nearly nonexistent. Why is that?

To figure it out, I had a two-hour discussion with

Soham Sankaran

, the founder and CEO of

PopVax

, an mRNA vaccine development startup based in Hyderabad. Amongst those in the know, Soham is well understood as one of the most talented biotech founders in India, and his company has had a genuinely incredible underdog success story. This story is still being written, but there's good reason to be bullish, given that PopVax has

an (in mouse) influenza vaccine that is 250x better than its competitors,

multiple large research collaborations

, and their

first upcoming US based phase 1 clinical trial being fully sponsored and conducted by the NIH

.

We discuss so many things. Including policy prescriptions for Indian R&D, why PopVax's vaccines are so good, how machine-learning is changing vaccine development, and much more. Links below!

Substack: https://owlposting.com/p/how-do-you-make-a-250x-better-vaccine

Youtube: https://youtube.com/watch?v=CHokQ5dMxHQ

Jargon explanation: https://owlposting.com/p/how-do-you-make-a-250x-better-vaccine?open=false#%C2%A7jargon-explanation

Transcript: https://owlposting.com/p/how-do-you-make-a-250x-better-vaccine?open=false#%C2%A7transcript

Apple Podcasts: https://podcasts.apple.com/us/podcast/how-do-you-make-a-250x-better-vaccine-at-1-10-the/id1758545538?i=1000688682418

Spotify: https://open.spotify.com/episode/5WcFoMDkxpQt4ULT1k5gm4
Less Laptop Velcro
by jefftk — 2025-02-09T03:30:03.403Z

A year ago I broke my laptop screen, and took the opportunity to build
something I've always wanted: a monitor that

folds vertically so I

don't have to bend my
neck:

A few months ago my cracked-screen laptop finished dying, and I got a
new one. I use the stacked monitor a lot less now, since for quick
things the built-in monitor is much more handy. But it's still
something I want when traveling, where I might be working on the
laptop for a significant time.

I originally used quite a lot of velcro on my laptop:

This turned out to be overkill, and to get worse over time: after a
few hours joined the bond was so secure that if my screen were not
already busted I would have been worried I'd crack it.

With the new laptop I put on just a tiny bit of velcro:

This turns out to be plenty: I haven't had any issues with it falling
off, but I can still detach it without feeling like I'm putting
excessive force on the machine.

Comment via: facebook, mastodon, bluesky
AXRP Episode 38.7 - Anthony Aguirre on the Future of Life Institute
by DanielFilan — 2025-02-09T01:10:05.683Z

YouTube link

The Future of Life Institute is one of the oldest and most prominant organizations in the AI existential safety space, working on such topics as the AI pause open letter and how the EU AI Act can be improved. Metaculus is one of the premier forecasting sites on the internet. Behind both of them lie one man: Anthony Aguirre, who I talk with in this episode.

Topics we discuss:

Anthony, FLI, and Metaculus

The Alignment Workshop

FLI’s current activity

AI policy

Work FLI funds

Daniel Filan (00:09):
Hello, everyone. This is one of a series of short interviews that I’ve been conducting at the Bay Area Alignment Workshop, which is run by FAR.AI. Links to what we’re discussing, as usual, are in the description. A transcript is, as usual, available at axrp.net. And, as usual, if you want to support the podcast, you can do so at patreon.com/axrpodcast. Well, let’s continue to the interview.

(00:29):
All right, well, I am currently speaking with Anthony Aguirre.

Anthony Aguirre (00:32):
Nice to be here.

Anthony, FLI, and Metaculus

Daniel Filan (00:33):
For those who aren’t as familiar with you and what you do, can you say a little bit about yourself?

Anthony Aguirre (00:39):
I’m a theoretical physicist by original training. I still hold a professorship at UC Santa Cruz, and I’ve studied a whole range of theoretical physics things in the early universe and gravity and foundations of quantum mechanics and all sorts of fun stuff. But about 10 years ago or so, I, as well as another colleague, Max Tegmark, and a few others, started to think about the long-term future, and in particular, transformative technologies and the role that they would be playing, and decided to start up a non-profit thinking about transformative technologies and what could be done to steer them in maybe a more favorable direction. At that point, AI was just starting to work a little bit and things like AGI were clearly decades away, if not centuries in some people’s opinion. So this was a more theoretical and pleasantly abstract set of things to think about.

(01:36):
But nonetheless, we felt like it was coming and started a bunch of activities and that was the birth of the Future of Life Institute. And I’m now, 10 years later, executive director of the Future of Life Institute pretty much full-time. And so I’ve got sort of a split between Future of Life Institute, my academic a...
[Job ad] LISA CEO
by Ryan Kidd — 2025-02-09T00:18:35.254Z

Overview

Job Title: Chief Executive Officer
Company Name: London Initiative for Safe AI (LISA)
Location: Old Street, London
Duration: Full-time
Salary Range: £95-125k (more may be available for an exceptional candidate)
Application Deadline: Monday 24th February

We are looking for a Chief Executive Officer with experience as a senior manager, start-up founder, or executive to lead LISA's high-performing, agile team in London. Interest in AI safety, especially large-scale and catastrophic risks from AI, is critical; previous experience of the field or funding ecosystem is desirable.

About LISA

The London Initiative for Safe AI (LISA) is a London-based charity (with UK Charity registration number: 1211693) dedicated to advancing education in artificial intelligence (AI) safety for the public benefit (including through research), and contributing to the safe development, use, governance, and regulation of AI.

Currently, LISA’s primary project is an AI safety co-working space in London, which hosts a wide range of researchers, programmes, and events. This office currently houses resident organisations (such as Apollo Research and BlueDot Impact), programmes (such as ARENA, the MATS extension, Pivotal Research, LASR Labs, PIBBSS Fellowship, and Catalyze Impact), and individual researchers affiliated with more than 50 other organisations and universities.

LISA aims to provide members with the resources, community, and support needed to positively impact the safe development and deployment of advanced AI systems. Please see our theory of change for more details.

About the role

Responsibilities

Lead LISA in its mission of improving the safety of advanced AI systems and growing the London AI safety community, though education and research (broadly defined);
Enhance and maintain LISA’s reputation as a leading home for AI safety research in London by developing LISA’s future vision, deciding and prioritising amongst competing opportunities to shape its growth;
Lead a team of (currently) five full-time staff, responsible for their performance/duties, well-being, and professional development;
Raise funds for LISA’s operations, widening the donor base and developing revenue streams. Produce strategy documents and impact analyses;
Develop LISA’s relationships with key stakeholders, including funders, office member organisations, external partners and key commercial suppliers;
Coordinate

...