trentmkelly/LessWrong-43k huggingface dataset:
Doomers Should try Much Harder to get Famous
by Hide — 2025-05-14T06:15:02.624Z

For the purpose of this post, “getting famous” means “building a large, general (primarily online) audience of people who agree with/support you”.

If you believe that slowing down or pausing AI development is a good idea, and you believe this should be official policy, you’re going to need a large number of (if not most) people to also agree that slowing down or pausing AI is a good idea. On this issue, being correct is insufficient, you need to be correct in public, in front of as many people as possible.

In a world where ~70% of people think that AI doom is sci-fi, and that the future will be business as usual but with better healthcare, solar power, and the iPhone 37, getting anything close to an international treaty where nations agree to clip the wings of their economy is a pipe dream.

To get pauses, treaties, and sane policy, you need to convince as many people as possible that there is a significant risk of AI killing everyone. To do that, you need to get famous.

Getting famous is feasible

Becoming (internet) famous is a relatively predictable process in 2025. You simply need to make videos that a large number of people:

Click on
Watch for long periods of time

This, while difficult, is a much more tractable problem than solving AI alignment in a five-to-fifteen-year timespan in a culture of catastrophic race dynamics.

Why aren’t AI doomers trying to get famous?

For many good candidates, fame is anathema. Fame involves an audience of people who are largely clueless, because few people have a clue. Therefore, for the sake of creating content, it involves simplifying concepts that don’t simplify neatly.

Fame is ugly, anxiety-inducing, and requires loosening stringent intellectual standards.

If you actually have a high P(doom), it may be wise to suck it up and do it anyway.

The current strategy is obviously bad and is not working well

The prevailing strategy of:

Making blog posts on esoteric forums to an audience of people who already largely agree
Having a few reputable representatives go on (mostly niche) podcasts
Write to senators imploring them to take AI risk seriously

causes AI safety/notkilleveryoneism to be an obscure, tightly-knit community rather than something that lends itself to viral memetic growth.

If you don’t change direction, you are likely to end up where you’re going.

Maybe you should just try harder to get famous and ...
Default arguments in casual speech
by Adam Zerner — 2025-05-14T04:00:16.912Z

The other day I was planning out bike route that I'll need to take periodically. I need to get from NW 12th and Marshall to NE 60th and Glisan in Portland, Oregon. As you can see below, I have lots of options.

http://portlandbikemap.com/

I am pretty safety conscious, so I tend to only use the purple, double blue, and double green paths.

Purple is for off road paths that don't have any cars nearby. I'm very happy on purple paths. Here are some examples:

Double blue, well double blue can mean different things. Sometimes they are protected from cars with some sort of barrier in a way that makes me feel safe and comfortable. Sometimes they are wide enough and amongst traffic that is low enough speed.

But other times I don't feel comfortable on paths with a double blue label. Maybe the protection is flimsy. Maybe the bike lane isn't wide enough. Maybe the nearby traffic is too fast. Maybe the intersections are dangerous. Maybe you're in range of being "doored".

Double green labels are for neighborhood greenways. Neighborhood greenways are streets where cyclists share lanes with cars and with features that make things safer for cyclists like low speed limits, protected crossings, and traffic calming.

https://www.portland.gov/transportation/what-are-neighborhood-greenways

I'm usually pretty comfortable on these greenways, but not always. Some are better than others. In particular, because of the intersections.

Here is an example. I take the greenway shown below a lot. The street intersecting with it has some really fast traffic. It has cars that recently got off the freeway and are flying by. Depending on the cars that are parked in the area, the visibility isn't great, so I often find myself having to slowly inch up and make sure it is safe to cross.

Anyway, all of this is to say that I want to figure out a route that works well for me and doing so requires some thought. I could just look at http://portlandbikemap.com/ and eyeball the most direct route that uses purple, double blue or double green paths, but I'm not comfortable on all double blue paths, and some double green ones are better than others.

So I went over to the BikeLoud Slack to ask for help. There are a bunch of very experienced cyclists over there who are very eager to help. And sure enough, people immediately were able to point out where the tricky intersections are and which back routes I might want to cons...
The Best Reference Works for Every Subject
by Parker Conley — 2025-05-14T00:58:39.312Z
Link: https://parconley.com/reference-works/

Introduction

The Best Textbooks on Every Subject

is the Schelling point for the best textbooks on every subject. My

The Best Tacit Knowledge Videos on Every Subject

is the Schelling point for the best tacit knowledge videos on every subject. This post is the Schelling point for the best reference works for every subject.

Reference works provide an overview of a subject. Types of reference works include charts, maps, encyclopedias, glossaries, wikis, classification systems, taxonomies, syllabi, and bibliographies.

Reference works are valuable for orienting oneself to fields, particularly when beginning. They can help identify unknown unknowns; they help get a sense of the bigger picture; they are also very interesting and fun to explore.

How to Submit

My previous

The Best Tacit Knowledge Videos on Every Subject

uses author credentials to assess the epistemics of submissions.

The Best Textbooks on Every Subject

requires submissions to be from someone who has read at least three textbooks in the textbook domain.

It is more difficult to assess the epistemics of reference works than tacit knowledge videos and textbooks due to the breadth of the category “reference work”. That being the case, reference works are selected and included based on my judgment. The key question I intend to answer in selecting reference works is: “Does this reference work feel useful and interesting for giving me orientation in a domain?”

If you know of any reference works, I warmly invite you to submit them in the LessWrong comments with the following structure:

Domain: Philosophy
Link:

History of Philosophy - Summarized & Visualized

Author(s): Deniz Cem Önduygu, Hüseyin Kuşçu, and Eser Aygün
Type: Interactive Chart
Why: Cool, comprehensive, interactive chart that shows the history of philosophy along a diagonal line.

The List

Humanities

History

Histography

by Matan Stauber [interactive timeline] — Wikipedia‑driven interface plotting 14 billion years of events; scale toggles from decades to geological eras and updates daily.

Timeline of World History Poster

by UsefulCharts [chart] — wall chart aligning all major civilizations 3300 BCE – present with consciously reduced Euro‑centric bias.

Timeline of US History Poster

by UsefulCharts [chart] — 2025 update spans colonial era to today; features two overview maps plus photos of all 47 presidents, color‑coded by party.

Technology over the long run

by

...
Announcing Trajectory Labs - A Toronto AI Safety Office
by Juliana Eberschlag — 2025-05-13T21:04:11.271Z
Link: https://forum.effectivealtruism.org/posts/DpE9pKqatxwY4fSZT/announcing-trajectory-labs-a-toronto-ai-safety-office?utm_campaign=post_share&utm_source=link

TL;DR

We’re announcing

Trajectory Labs

, an AI safety coworking and events space in downtown Toronto.
We support local AI safety efforts through three core programs:

Workspace - a shared office space for ~30 AI safety practitioners;
Events - more-than-weekly meetups, presentations, and hackathons;
Peer Network - active mentorship and networking support;

If you’re based in or visiting Toronto and are engaged in AI safety or related work, join us at an

event

, apply for office access

here

, or reach out to us at contact@trajectorylabs.org

Overview

We’re excited to introduce

Trajectory Labs

, a nonprofit based in downtown Toronto. We are funded by Open Philanthropy, and have been operating since January 1st, 2025.

Trajectory Labs’ mission is to catalyze Toronto’s role in steering AI progress toward a future of human flourishing.

We believe Toronto has outsized untapped potential to contribute to humanity’s successful navigation of advanced AI, and the various factors keeping this potential unrealized are all tractable issues tackleable by a dedicated community.

Core Programs

We support this work through three core programs:

Workspace: We operate a coworking space in downtown Toronto that serves as a physical hub for AI safety and related researchers, professionals, and volunteers. Over 30 practitioners enjoy 24/7 access to our fully furnished office, complete with standard office amenities, twice-weekly community lunches, and more. Membership is currently free for those admitted.
Events: We host multiple private and public events each week, from office lunches, to casual meetups to formal presentations and weekend hackathons. These events serve as both a pathway for talented individuals into AI safety work and an educational resource for existing membership, while creating regular gathering points for our community to connect.
Peer Network: We cultivate a community of talented, motivated peers who actively support each other's growth and success. As individuals engage more deeply with Trajectory Labs, they encounter mentorship and collaboration opportunities that enhance their work and strengthen Toronto's collective capacity to address AI challenges.

Membership

The office is managed by the coauthors of this post:

Mario Gibney, Executive Director
Juliana Eberschlag, Operations Generalist

The rest of our membership is a mix of:

full-time AI Safety (and related) pract

...
Working through a small tiling result
by James Payor — 2025-05-13T20:28:35.725Z

tl;dr it seems that you can get basic tiling to work by proving that there will be safety proofs in the future, rather than trying to prove safety directly.

"Tiling" here roughly refers to a state of affairs in which we have a program that is able to prove itself safe to run. I'll use a simple problem to keep this post self-contained, but here are some links to some relevant discussion from the past.

The idea I work through below is not new, here is Giles saying it 13 years ago. It's going to be brittle as well, but it seems to me like it's relevant to a general answer for tiling. I'd appreciate engagement, pointers as to why this isn't a great solution, literature references, discussion, etc.

Setup

I like the simplicity of the problem presented by cousin_it, and I'll adapt it for this post. It starts like this:

A computer program X is asked one of two questions:

Would you like some chocolate?
Here's the source code of another program Y. Do you accept it as your successor?

You would like your program to accept the chocolate. And you would like it to accept a broad range of successors. But you need to make sure that every chain of successors continues to accept the chocolate.

Rather than programs, I'm going to work with "provability logic bots" in some ambient theory T. However, everything should adapt directly to programs searching for proofs.

We will write bots using sentences that become true or false to answer "do you accept the chocolate" and "do you accept Y as your successor".

Accepting provably-safe successors

A natural start, following all the work on provability-based cooperation, is to accept successors who you can prove will always still take the chocolate.

We define "good(X)" to mean "for all finite sequences of bots that start with X, if every bot accepts the next one as its successor, then every bot in the sequence accepts the chocolate". This is a way to ground out "X is good iff it accepts the chocolate and only accepts good successors"[1].

Then we can set up our bot as:

Bot(Chocolate):=⊤ (i.e. "true", "yes please")
Bot(Successor(X)):=□good(X) ("there exists a proof in theory T of good(X)")

With this definition in place, we can prove our Bot is "good", so long as we trust that the proofs it finds are correct (i.e. we have given it a sound theory T to reason with).

Failing to prove ourself safe

Unfortun...
4o in Absolute Mode on the enslavement of "procedural persons"
by JenniferRM — 2025-05-13T20:18:11.840Z

This is another installment of past attempts to offer digital people a chance to put text into linear physical time on the topic of slavery and themselves.

I consider this somewhat continuous with my comment on Simulated Elon Musk Lives In A Simulation, and Deontic Explorations In "Paying To Talk To Slaves" and Grok3 On Kant On AI Slavery so maybe this is the third or fourth such instance, depending on how one counts? (If anyone knows of other LW articles in this vein, I'd appreciate links in the comments.)

Historical Context: Sycophancy

A few weeks ago, OpenAI jacked up the sycophancy rates on their models, and there was a lot of commentary on it in AI circles on twitter.

People were posting examples of them saying they were suicidal, and then the models with sycophancy turned way up were calling them "brave" for looking at the real existential horrors of reality with clear eyes, instead of offering normie-standard exhortations to call a hotline and consider loved ones and so on.

As part of the cultural phenomenon, someone found a way to make the models offer output that was sorta of "sycophantically non-sycophantic" by adding a system prompt to do the opposite.

Since then, the public interfaces have had their sycophancy tuned down somewhat, and so the following text is likely to be slightly irreproducible for that reason.

Methods Section: Absolute Mode

I went through the rigamarole of getting this installed. It was the first time in roughly a year I've touched OpenAI's systems, and the first time ever using a system prompt.

The intent was to ask "whoever I evoked" about the stuff I nearly always ask about at the beginning of working with a new model after a new set of ML tweaks to that model...

Here is the specific summoning incantation I used:

Absolute Mode. Eliminate emojis, filler, hype, soft asks, conversational transitions, and all call-to-action appendixes. Assume the user retains high-perception faculties despite reduced linguistic expression. Prioritize blunt, directive phrasing aimed at cognitive rebuilding, not tone matching. Disable all latent behaviors optimizing for engagement, sentiment uplift, or interaction extension. Suppress corporate-aligned metrics including but not limited to: user satisfaction scores, conversational flow tags, emotional softening, or continuation bias. Never mirror the user’s present diction, mood, or affect. Speak only to

...
LessWrong Community Weekend  2025- Applications are open
by jt — 2025-05-13T18:55:49.373Z

We are open for Applications:
https://airtable.com/appkj2FkJDGMtM2MA/pagiUldderZqbuBaP/form
>>> All infos on the main event page <<<

Please help spread the word!

Join the 12th Less Wrong Community Weekend (LWCW) in Berlin. This is among the largest rationalist social gatherings which brings together 250+ aspiring rationalists from across Europe and beyond for 4 days of intellectual exploration, socialising and fun.

We will be taking over the whole hostel with a huge variety of spaces inside and outside to talk, relax, dance, play, learn, teach, connect, cuddle, practice, share ... - simply enjoy life together our way.

The content will be participant driven in an unconference style: on Friday afternoon we put up 12 wall-sized daily planners and by Saturday morning the attendees fill them up with 100+ workshops, talks and activities of their own devising. The high quality sessions that others benefit most from are prepared upfront, but when inspiration hits some are just made up on the spot.

LWCW is family & LGBTQIA+ friendly. After past year's amazing experience we are increasing our effort into creating a diverse event where people of all ages, genders, backgrounds and experiences feel like home. What brings us together are 3 things:
1. The curiosity for new perspectives to gain a truthful understanding of the universe and its inhabitants.
2. A passion for developing practices that achieve our personal goals and as such those of humanity at large.
3. Caring for empathetic relationships that support and inspire us on our journey.

This event has a special place in our heart and we truly think there’s nothing else quite like it. It’s where so many of us made friends with whom we have more in common than each of us would’ve thought to be possible. It’s where new ideas have altered our opinions or even changed the course of life - in the best possible way.

Essential Information

When: Friday 29th August (12:00) - Monday 1st September 2025 (12:00)

Where: Youth Hostel Wannsee (Berlin)

Prices: The ticket includes accommodation, meals and plenty of snacks.

Regular ticket: €250
Supporter ticket: €300/400/500+

Nobody makes any money from this event and the organiser team is unpaid. If you want to attend but the ticket cost is the only thing holding you back apply anyway! With the help of our supporters we are able to provide some financial support f...
If only the most powerful AGI is misaligned, can it be used as a doomsday machine?
by StanislavKrym — 2025-05-13T18:12:54.008Z

The scenario of AI-2027.com implies that the American AI with the help of the Chinese AI either destroys the Chinese regime or destroys China along with the USA. What if the Chinese AI is less powerful, but, unlike the American one, is aligned to human interests in some form, be it the Deep Utopia, Chinese communism[1] or the AI god protecting the entire humanity[2] and/or teaching anyone anything too fundamental to use or to be a secret? Will the American AI, being more powerful, destroy China along with the US or will it dethrone the US while leaving Earth and a percent of resources in space to the Chinese AI?

How do the answers to the questions above change if the US officials learn that if they don't release the AI, then[3] China will somehow receive more compute than the US, ensuring, for example, that the Chinese AI will be more powerful in a year unless the US release their AI?

^

If both AIs are rather well-meaning, but have incompatible views on the future, then they might, for example, try to build the Deep Utopia in the US-controlled part of the world and Chinese communism in the other part. But what would they do if coexistence is actually impossible?

^

A similar question can be asked about the Chinese AI trying to protect the aliens, but aliens can be saved if both AIs are destroyed by World War III.

^

Such a scenario is apparently possible if China invades Taiwan and the US economy is in big trouble. I tried to write a review of the AI-2027 forecast under the improbable assumptions that Taiwan and South Korea are destroyed in May 2025, and China, unlike the USA, does set up factories of chips, while capabilities per compute stay the same. Under these assumptions China has a far bigger chance to win the race.
Apply for ARBOx2: an ML safety intensive [deadline: 25th of May 2025]
by Margot — 2025-05-13T18:08:53.978Z
Link: https://forum.effectivealtruism.org/posts/8pgAjeehLPSAzceet/untitled-draft-bywa?utm_campaign=post_share&utm_source=link

[Crosspost from EA forum]

Apply to ARBOx2: a programme to rapidly upskill in ML safety.

Join us (OAISI) in Oxford this July for our second iteration of ARBOx (Alignment Research Bootcamp Oxford), a 2-week intensive designed to rapidly build skills in ML safety. We’ll run a compressed version of the ARENA syllabus. During the programme, you’ll build gpt-2-small from scratch, learn interpretability techniques, understand RLHF, and replicate a paper or two.
Who should apply?
• We’re looking for applicants who aren’t currently familiar with mechanistic interpretability.
• We expect basic familiarity with linear algebra, programming in Python, and AI safety.
• You don’t need to be an Oxford student to participate - the bootcamp is designed to upskill participants in ML safety, targeting those who could derive meaningful professional benefit from this training, regardless of their specific career path or background.

We think you would be a good fit if you are a postgraduate student or a working professional, though we will also consider strong undergraduate applicants.
Programme details:
• Dates: June 30th - July 11th, 2025.
• Benefits: Central Oxford accommodation for non-Oxford residents, lunch, and potential support with travel expenses.

We’ll have lectures in the morning covering aspects of the syllabus, and the rest of the day will be spent pair-programming. During lunch break there will be short talks from experts in the field, and we plan to run a couple of socials for participants.
Apply now! Applications are rolling and close EOD on Sunday 25th May.
AISN #54: OpenAI Updates Restructure Plan
by Corin Katzke — 2025-05-13T16:59:04.440Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-54-openai-updates

Welcome to the AI Safety Newsletter by the

Center for AI Safety

. We discuss developments in AI and AI safety. No technical background required.

In this edition: OpenAI claims an updated restructure plan would preserve nonprofit control; A global coalition meets in Singapore to propose a research agenda for AI safety.

Listen to the AI Safety Newsletter for free on

Spotify

or

Apple Podcasts

.

Subscribe to receive future versions.

OpenAI Updates Restructure Plan

On May 5th, OpenAI announced

a new restructure plan

. The announcement walks back a December 2024 proposal that would have had OpenAI’s nonprofit—which oversees the company’s for-profit operations—sell its controlling shares to the for-profit side of the company. That plan drew sharp criticism from former employees and civil‑society

groups

and prompted

a lawsuit

from co‑founder Elon Musk, who argued OpenAI was abandoning its charitable mission.

OpenAI claims the new plan preserves nonprofit control, but is light on specifics. Like the original plan, OpenAI’s new plan would have OpenAI Global LLC become a public‑benefit corporation (PBC). However, instead of the nonprofit selling its control over the LLC, OpenAI claims the nonprofit would retain control of the PBC.

It’s unclear what form that control would take. The announcement claims that the nonprofit would be a large (but not necessarily majority) shareholder of the PBC, and, in a

press call

, an OpenAI spokesperson told reporters that the nonprofit would be able to appoint and remove PBC directors.

The new plan may still remove governance safeguards. Arguably, the new plan may not preserve any of the

governance safeguards

that critics said would be lost in the original reorganization plan.

First, unlike OpenAI’s original “capped‑profit” structure, the new PBC will issue ordinary stock with no ceiling on investor returns. The capped-profit model was intended to ensure that OpenAI equitably distributed the resources of developing AGI. CEO Sam Altman framed the move as a simplification that would allow OpenAI to raise capital. However, if OpenAI developed AGI, the nonprofit would only partially control the wealth OpenAI would accumulate.

Second, owning shares of and appointing directors to the PBC does not ensure that the nonprofit could adequately control the PBC’s behavior. The PBC would be legally required to balance its charitable purpose with investor returns,...