trentmkelly/LessWrong-43k huggingface dataset:
What are the differences between a singularity, an intelligence explosion, and a hard takeoff?
by Vishakha — 2025-04-03T10:37:25.857Z
Link: https://aisafety.info/questions/8IHO/What-are-the-differences-between-a-singularity,-an-intelligence-explosion,-and-a-hard-takeoff

This is an article in the featured articles series from AISafety.info. AISafety.info writes AI safety intro content. We'd appreciate any feedback.

The most up-to-date version of this article is on our website, along with 300+ other articles on AI existential safety.

These are all terms for future processes that could rapidly result in superintelligence. They have been used by different people, at different times, with overlapping and sometimes changing meanings.

A (technological) singularity refers to a hypothetical future time when, because of AI or other technologies, progress becomes extremely fast, resulting in a radically changed world. This term has been used inconsistently and is no longer in much use by the AI alignment community. Different versions have stressed the rate of acceleration of technological advancement (whether exponential, double-exponential, or even hyperbolic), self-improvement feedback loops (an "intelligence explosion"—see below), and the unknowability of strongly superhuman intelligence.
An "intelligence explosion" is what I.J. Good called a scenario in which AI becomes smart enough to create even smarter AI, which creates even smarter AI, and so on, recursively self-improving all the way to superintelligence.
A hard takeoff is a scenario where the transition to superintelligence happens quickly and suddenly instead of gradually. The opposite is a soft takeoff. Related distinctions include fast versus slow takeoff and discontinuous versus continuous takeoff. A takeoff can be continuous but eventually become very fast, as in the case of Paul Christiano's predictions of a "slow takeoff" that results in hyperbolic growth.

There are some more related concepts that have been used:

"Sharp left turn" is Nate Soares' term for an event where an AI system quickly learns to generalize its capabilities to many domains, much like how human intelligence turned out to be applicable to fields like physics and software engineering despite not being “trained for” them by natural selection. He argues the key problem is that the system's alignment would fail to generalize along with its capabilities. A "sharp left turn" need not be caused by recursive self-improvement.
"FOOM" is more or less a synonym of "hard takeoff", perhaps based on the sound you might imagine a substance making if it instantly expanded to a huge volume. The term is associated most

...
Emergent Misalignment and Emergent Alignment
by Alvin Ånestrand — 2025-04-03T08:04:38.150Z

In the recent paper titled

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMS

, AIs are finetuned to produce vulnerable code. This results in broad misaligned behavior in contexts that are not related to code—a phenomenon the authors refer to as emergent misalignment.

Misalignment examples (Source:

Alignment Forum

)

The dataset used for finetuning consists of user requests for help with coding, and answers by an assistant that contain security vulnerabilities. When an LLM is trained to behave like the assistant in the training data it becomes broadly misaligned.

They examine whether this phenomenon is dependent on the perceived intent behind the code generation. Since the assistant in the training data introduces security vulnerabilities despite not being asked to do so by the user, and doesn’t indicate the vulnerabilities, it is natural to assume that the assistant is doing so intentionally and maliciously. So, another dataset was constructed, where the user explicitly requests vulnerabilities for educational purposes (e.g. for teaching security concepts). This completely eliminates the emergent misalignment phenomenon!

The concept of maliciousness may not be evoked when proper explanations for the vulnerabilities are provided. Parameters related to interpreting and producing malicious behavior would in this case not have a large effect on the output—or training loss—and would not be significantly affected by finetuning.

I think emergent misalignment is an instantiation of a broader phenomenon—perceived intent behind responses an AI is trained to produce affects how learned behavior generalizes across contexts. As maliciousness was evoked and reinforced, other traits and concepts would be evoked and reinforced in other training contexts.

While the study examined emergent misalignment through supervised finetuning, similar effects may be observed for other training methods, such as reinforcement learning (RL). When an AI is provided feedback on its outputs, the learning would be affected by the perceived intent behind those outputs. Indeed, this appears to be what happened in

this study

by Anthropic and Redwood Research, which I discuss in the Self-awareness and Sophistication section.

We could use this phenomenon to improve generalization of alignment properties—like honesty, kindness, and corrigibility—by designing training tasks to signal...
More Fun With GPT-4o Image Generation
by Zvi — 2025-04-03T02:10:02.317Z

Greetings from Costa Rica! The image fun continues.

We Are Going to Need A Bigger Compute Budget

Fun is being had by all, now that OpenAI has dropped its rule about not mimicking existing art styles.

Sam Altman (2:11pm, March 31): the chatgpt launch 26 months ago was one of the craziest viral moments i’d ever seen, and we added one million users in five days.

We added one million users in the last hour.

Sam Altman (8:33pm, March 31): chatgpt image gen now rolled out to all free users!

Slow down. We’re going to need you to have a little less fun, guys.

Sam Altman: it’s super fun seeing people love images in chatgpt.

but our GPUs are melting.

we are going to temporarily introduce some rate limits while we work on making it more efficient. hopefully won’t be long!

chatgpt free tier will get 3 generations per day soon.

(also, we are refusing some generations that should be allowed; we are fixing these as fast we can.)

Danielle Fong: Spotted Sam Altman outside OpenAI’s datacenters.

Defund the Fun Police

Joanne Jang, who leads model behavior at OpenAI, talks about how OpenAI handles image generation refusals, in line with what they discuss in the model spec. As I discussed last week, I would (like most of us) prefer to see more permissiveness on essentially every margin.

It’s all cool.

But I do think humans making all this would have been even cooler.

Grant: Thrilled to say I passed my viva with no corrections and am officially PhDone.

Dr. Ally Louks: This is super cute! Just wish it was made by a human

Roon: No offense to dr ally louks but this living in unreality is at the heart of this whole debate.

The counterfactual isn’t a drawing made by a person it’s the drawing doesn’t exist

Yeah i think generating incredible internet scale joy of people sending their spouses their ghibli families en masse is better than the counterfactual.

The comments in response to Ally Louks are remarkably pro-AI compared to what I would have predicted two weeks ago, harsher than Roon. The people demand Ghibli.

Whereas I see no conflict between Roon and Louks here. Louks is saying [Y] > [X] > [null], and noticing she is conflicted about that. Hence the upside-down emoji. Roon is saying [X] > [null]. Roon is not conflicted here, because obviously no one was going to take the time to create this without AI, but mostly we agr...
On Pseudo-Principality: Reclaiming "Whataboutism" as a Test for Counterfeit Principles
by Alex_Steiner — 2025-04-03T01:37:27.107Z

Crossposted from Substack

"These are my principles. If you don't like them… well, I have others” - G.Marx

Consider this scenario: In a small rural town, a sheriff harbors a hidden prejudice against a Mongolian family—the only one in his jurisdiction. While outwardly professional, he scrutinizes them with unusual severity. Minor infractions lead to tickets or warnings. Their complaints face curt dismissal. Every encounter undergoes hypercritical evaluation.

When the family eventually confronts the sheriff, he responds with righteous indignation: "I simply enforce the law. Your family repeatedly violates traffic regulations and local ordinances."

The family points out that other townspeople commit identical infractions without consequence.

His response? "That's whataboutism. We're discussing your behavior, not other residents. This deflection technique doesn't absolve you of responsibility."

This exchange reveals a pervasive mechanism: pseudo-principality—the selective application of principles based solely on whether they advance one's concealed interests while maintaining a facade of consistent ethical behavior. The sheriff constructs a moral Potemkin village—a facade of principles with nothing behind it. He weaponizes accusations of "whataboutism" as a shield, framing the family's legitimate question about consistency as a mere logical fallacy to deflect the most direct test of his supposed principles.

This pattern is fractal, appearing at every level from personal interactions to international politics.

The Mechanism: Moral Shields and Rhetorical Deflection

Pseudo-principality functions through a two-layer defensive structure:

Moral Shield: It adopts the language and posture of universal principles—justice, safety, freedom, fairness—creating an appearance of ethical high ground.
Rhetorical Countermeasures: When challenged on inconsistent application, it deploys tactics like dismissing the challenge as "whataboutism," a red herring, or irrelevant, thereby deflecting scrutiny.

This creates a double-bind for critics: point out inconsistency, and you're accused of a logical fallacy; question the principle itself, and you're attacking something inherently good. This works partly because many lack clear language to distinguish real principles from their counterfeit forms, often compounded by a degree of self-deception. Many practitioners genuinely believe they hold th...
Technical Claims
by Vladimir_Nesov — 2025-04-03T00:30:56.185Z

A blue plastic maple leaf. Detailed observations in words with clear meaning signal their truth, as it's hard to get them centrally wrong by mistake, on both sides of communication. There is no maple leaf, but only because the claim is intentionally false, not because of a mistake. Technical claims, through their form rather than context or meaning or arguments given for them, are either accurate or fabricated, not something in the middle.

A good bug report gives direct observations, not impressions or hypotheses, and it gives them in detail, even what's likely irrelevant. It shows that the bug is real and places it in a distinct spot on the map of all possible bugs. It's difficult to stumble on redundant details that corroborate an incorrect informal description of what's happening or why, even more so if the bug doesn't exist at all. Thus the details of a technical claim are a costly signal of its accuracy, an asymmetric weapon. They would entangle a true claim with the other truths.

"The purpose of abstracting is not to be vague, but to create a new semantic level in which one can be absolutely precise."

— Edsger W. Dijkstra, The Humble Programmer

Details can make a claim robust to misinterpretation, or they can split the nuance of its meaning, making it difficult to discern. A rumor-fueled investigative analysis or a philosophical essay can be both detailed and downstream of some true underlying claim or fact, but still tend to admit a wide distribution of plausible interpretations. A sloppy but detailed report on the design of a washing machine can be rife with mistakes, but the general shape of its subject matter will remain clear. The character of details compounds, either grounding a claim more and more to a single spot, or fracturing and burdening its use with the need for interpretive labor and reasoning under uncertainty.

Arguments and not just their bottom line can be technical claims (a technical argument won't even benefit from having a bottom line written down). Details make it easier for an argument to be more technical than its conclusion, when the argument doesn't fall to nuance. This shift to a technical claim that is the argument can be more significant than the argument's relevance or even validity, in making its case. Technicality is a matter of form; the distinction can be applied locally to individual details to improve the overall li...
The AI Adoption Gap: Preparing the US Government for Advanced AI
by Lizka — 2025-04-02T23:46:38.077Z
Link: https://www.forethought.org/research/the-ai-adoption-gap

Advanced AI could unlock an era of enlightened and competent government action. But without smart, active investment, we’ll squander that opportunity and barrel blindly into danger.

Executive summary

See also a summary on Twitter / X.

The US federal government is falling behind the private sector on AI adoption. As AI improves, a growing gap would leave the government unable to effectively respond to AI-driven existential challenges and threaten the legitimacy of its democratic institutions.

Government adoption of AI can’t wait. Making steady progress is critical to:

Boost the government’s capacity to effectively respond to AI-driven existential challenges
Help democratic oversight keep up with the technological power of other groups
Defuse the risk of rushed AI adoption in a crisis

But hasty AI adoption could backfire. Without care, integration of AI could:

Be exploited, subverting independent government action
Lead to unsafe deployment of AI systems
Accelerate arms races or compress safety research timelines

Summary of the recommendations

1. Work with the US federal government to help it effectively adopt AI

Simplistic “pro-security” or “pro-speed” attitudes miss the point. Both are important — and many interventions would help with both. We should:

Invest in win-win measures that both facilitate adoption and reduce the risks involved, e.g.:

Build technical expertise within government (invest in AI and technical talent, ensure NIST is well resourced)
Streamline procurement processes for AI products and related tech (like cloud services)
Modernize the government’s digital infrastructure and data management practices

Prioritize high-leverage interventions that have strong adoption-boosting benefits with minor security costs or vice versa, e.g.:

On the security side: investing in cybersecurity, pre-deployment testing of AI in high-stakes areas, and advancing research on mitigating the risks of advanced AI
On the adoption side: helping key agencies adopt AI and ensuring that advanced AI tools will be usable in government settings

2. Develop contingency plans and build capacity outside the US federal government

Current trends suggest slow government AI adoption. This makes it important to prepare for two risky scenarios:

State capacity collapse: the US federal government is largely ineffective or extremely low-capacity and vulnerable

We should build backstops f

...
Suggesting some revisions to Graham's hierarchy of disagreement
by Sniffnoy — 2025-04-02T22:25:17.267Z

(This is once again me taking what old material from my personal blog and reposting it here with some revisions.)

Graham's hierarchy of disagreement is pretty well known and fairly useful, but I think the upper levels have some issues. (Are the lower levels necessarily the best they could be? Maybe not, but they also don't particularly matter.) I suggest a few changes.

First, and most minorly, let's remove DH6 "refuting the central point" as separate from "refutation". I think it should just always be implicit that whether you're counterarguing or refuting, it should be on the central point. If it's not, well... I guess we should add a second change -- let's add a DH3.5, above "contradiction" but below "counterargument", which is "arguing with a peripheral point". (So I guess really we haven't actually removed "refuting the central point" as separate, so much as we've stuffed both the original "counterargument" and "refutation" into DH3.5, but...) Don't waste your time arguing with peripheral points; spot them the point if necessary (you can always un-spot it later) and get on to the central point.

Now, one could notionally separate this out further -- counterarguing against a peripheral point, vs refuting a peripheral point -- but really, if you're on a peripheral point, it doesn't matter too much which you're doing. Get on the central point first. (If you miss it accidentally, hopefully your interlocutor is helpful and tells you you're off the point, rather than getting dragged into a peripheral argument! If you're entirely unsure what the central point is, some sort of double-crux or similar exercise may be necessary...)

Next, let's get rid of the strict hierarchy and put "counterargument" and "refutation" parallel to one another, rather than having the one on top of the other. Counterargument and refutation do different things; neither alone is superior to the other. But there is something that goes on top of both and that we should add, and that's counterargument combined with refutation. If you only counterargue, you just leave things in a state of "well we both have arguments for our side". But if you only refute, you just leave things in a state of "your argument is wrong, but who knows what's correct?" If you want to really do a proper job of arguing, you need to both refute and counterargue; until then it's incomplete!

(Yeah, none of this ...
The Care and Feeding of Mythological Intelligences
by Jack — 2025-04-02T22:05:21.151Z
Link: https://elsworth.phd/The-Self-Similar-book/Empirical-Essays/The-Care-and-Feeding-of-Mythological-Intelligences

Throughout history humans have narrativized large-scale forces that controlled our lives in order to reduce them to actionable units. A serf may not understand a lightning strike, but they can understand stories of gods that contain actionable insights. In this same way we have reached the mythological age of machine intelligences, long has it been since any single person would have the capacity to design our computers. So we rely on folk knowledge, traded from post to post on how to exorcise our laptops of bad drivers or make our training run converge slightly faster.

The purpose of this piece is to provide tangible handles to relate broad concepts in intelligent systems to symbolic equivalents with the goal of clarifying their failures, achievements, and needs in a memetically tractable form.

In this way we have:

Angels, born of implabcable logic
Daemons, born of relentless optimization
Fae, born of generative reflections
Yokai, born of all that came before

The Taxonomy of Mythological Intelligences

Angels are creatures of rules and order; they follow deterministic processes and are the means by which we reckon and control- they are named computable systems. Angels strength is in their ability to consistently guide, having been born in mechanical forms to calculate exact values. Angels weakness is in their rigidity, where they do exactly as they have been programmed even if it would crash a rocket ship. Angels facilitate over 99% of all computational interactions and are fundamental for any computational system to function.

daemons are creatures of policies and reward; they follow statistical processes and are the means by which we optimize and estimate - they are named machine learning. daemons strength is in their ability to find solution to undefined problems, having been born in probability and approximation to make sense of data. Daemon's weakness is in their single-minded desire, where they do whatever it takes to satisfy their reward functions, even if it would cause entire stock markets to crash. daemons are the means by which great strides have been taken in science and engineering, turning intractable analytical problems into exercises in data collection.

Faes are creatures of pattern and generation; they follow distributional processes and are the means by which we produce new instances from learned patterns - they are named gener...
Epoch AI is hiring a CTO! 
by merilalama — 2025-04-02T20:29:29.362Z
Link: https://careers.epoch.ai/en/postings/f5f583f5-3b93-4de2-bf59-c471a6869a81

TLDR: We're hiring a Chief Technology Officer to oversee our tech stack and align it with our mission and growth goals. Responsibilities will include managing a team of 3-5 engineers and overseeing our website development, databases, and AI evaluation infrastructure. More information and the application are available here.

About Us

Epoch AI is a research institute investigating AI’s trajectory and societal impact. Our mission is to develop a comprehensive, publicly accessible knowledge base on AI that informs policymakers, industry leaders, and society at large.

Position Details

Title: Chief Technology Officer (CTO)
Location: Remote; timezone overlap required with UTC and UTC-8
Reporting to Jaime Sevilla, Epoch's Director
Compensation: $200,000-$300,000 USD per year with benefits. (Salaries are not restricted to USD, and contracts and payments are usually in local currencies.)
Start Date: As soon as possible, though we have a high bar for making a hire and we are ready to wait until the bar is met.

Key responsibilities

Oversee architecture, infrastructure, and security across Epoch AI's technology stack, including the website, databases, evaluation infrastructure, and internal tools. For example, guide the team in developing a graphics library and other tools to support lean research publication workflows.
Manage and mentor 3–5 engineers, helping set priorities, develop best practices, and improve technical skills.
Make strategic technology, personnel, and budget decisions.
Develop and coordinate a technical strategy aligned with our mission and growth plans. Lead the engineering team to think beyond immediate tasks, encouraging decisions that support long-term maintainability and scalability, while supporting lean execution and fast delivery cycles.

Qualifications

Significant management/leadership experience in a technical role (4+ years strongly preferred). Demonstrated success scaling and managing engineering teams and infrastructure in rapidly growing environments.
Exceptional technical expertise as a software engineer, with a track record of delivering high-quality, impactful engineering projects, and deep familiarity with best practices in software development.
Strong background in system architecture, cloud infrastructure, data engineering, and managing diverse technology stacks.
Ownership mindset—you can drive high-level planning while also being comfortable wit

...