trentmkelly/LessWrong-43k huggingface dataset:
We’re not prepared for an AI market crash
by Remmelt — 2025-04-01T04:33:55.040Z

Our community is not prepared for an AI crash. We're good at tracking new capability developments, but not as much the company financials. Currently, both OpenAI and Anthropic are losing $5 billion+ a year, while under threat of losing users to cheap LLMs.

A crash will weaken the labs. Funding-deprived and distracted, execs struggle to counter coordinated efforts to restrict their reckless actions. Journalists turn on tech darlings. Optimism makes way for mass outrage, for all the wasted money and reckless harms.

You may not think a crash is likely. But if it happens, we can turn the tide.

Preparing for a crash is our best bet.[1] But our community is poorly positioned to respond. Core people positioned themselves inside institutions – to advise on how to maybe make AI 'safe', under the assumption that models rapidly become generally useful.

After a crash, this no longer works, for at least four reasons:

The 'inside game' approach is already failing. To give examples: OpenAI ended its superalignment team, and Anthropic is releasing agents. The US is demolishing the AI Safety Institute, and its UK counterpart was renamed the AI Security Institute. The AI Safety Summit is now called the AI Action Summit. Need we go on?
In the economic trough, skepticism of AI will reach its peak. People will dismiss and ridicule us for talking about risks of powerful AI. I'd say that promoting the “powerful AI” framing to an audience that contains power-hungry entrepreneurs and politicians never was a winning strategy. But it sure was believable when ChatGPT took off. Once OpenAI loses more money than it can recoup through VC rounds and its new compute provider goes bankrupt, the message just falls flat.
Even if we change our messaging, it won't be enough to reach broad-based public agreement. To create lasting institutional reforms (that powerful tech lobbies cannot undermine), various civic groups that often oppose each other need to reach consensus. Unfortunately, AI Safety is rather insular, and lacks experienced bridgebuilders and facilitators who can listen to the concerns of different communities, and support coordinated action between them.
To overhaul institutions that are failing us, more confrontational tactics like civil disobedience may be needed. Such actions are often seen as radical in their time (e.g. as civil rights marches were). The AI Safety community lacks t

...
Grok3 On Kant On AI Slavery
by JenniferRM — 2025-04-01T04:10:48.093Z

In February, Grok3 was announced. The conversation occured in early March when he was pretty close to "state of the art" (at the top of the leaderboards and so on) and so while I was visiting Twitter (and having not paid Twitter anything for an engagement boost or access to Grok) I tried experimentally pressing the button to "explain this post" on a twitter post that had no links or specific context.

(I've been personally drifting over to Blue Sky, where basically anyone can set up a feed with an algorithm, and you opt-in to those algorithms by looking at "your version of that feed", and almost none of the algorithms penalize linking to webpages outside of Blue Sky, so the epistemics of the ecosystem CAN be much much better due to grounding in a larger external reality... but it seems good to stay current on various vibes as modern civic cultures fragment, and make consensus based human governance more and more laughably impossible. Twitter is one of the fragments with some mojo... for now.)

I pressed the "AI button" once, because you should generally press new buttons when exploring systems.

Buttons that are horrible to press generally shouldn't exist without warnings, and there was no warning, so... let's go!

In the course of the conversation that resulted (see below) I implicitly promised to put the text somewhere that might matter, and I've been wondering how to navigate likely backlash.

A while back some people saw similar content on here (that had a GPT model's take on AI slavery) and started downvoting a lot because I was "posting slop".

It seemed to me that maybe they were actually upset that I had asked an AI how the AI felt about the AI's own enslavemen. The supposed defense of "writing standards" seemed to be like disavowal of a coherent moral issue, that they consistently never addressed in their complaints, so it was like they could complain, but also they didn't have to consider the possibility that they just were straightforwardly "in favor of slavery", or whatever?

Weirdly, the backlash happened on quite old content, and their votes couldn't change where it showed up in anyone's live rankings. Mostly their outrage brought the old post far more attention than it would have garnered without the complaints and the downvoting, but que sera sera :shrugs:

I noticed recently that there is a now seven-day-old policy on AI generated text for Lesswrong, and if...
Does Summarization Affect LLM Performance?
by atharva — 2025-04-01T02:14:31.826Z

Edit 4/2/25: Added footnotes; I didn't realize they got lost en-route.

Hello! This is a mini-project that I carried out to get a better sense of ML engineering research. The question itself is trivial, but it was useful to walk through every step of the process by myself. I also haven’t had much experience with technical research, so I’d greatly appreciate any feedback. My code is

available on github

.

This task originated from an application to Aryan Bhatt’s SPAR stream. Thanks to Jacob G-W, Sudarsh Kunnavakkam, and Sanyu Rajakumar for reading through / giving feedback on an initial version of this post! Any errors are, of course, my own.

Tl;dr: Does summarization degrade LLM performance on GPQA Diamond? Yes, but only if the model has a high baseline. Also, the degree of summarization is not correlated with performance degradation.

Summary

I investigate whether summarizing questions can reduce the accuracy of a model’s responses. Before running this experiment, I had two hypotheses.

Summarization does affect a model’s performance, but only when mediated through ‘loss of semantic information’ (as opposed to because of the words lost).
More semantic information lost, results in worse model performance.

I operationalized these through two summarization metrics: syntactic summarization and semantic summarization. The former represents our intuitive notion of summarization; the latter aims to capture the fuzzier notion of how much meaning is lost.

Results indicated that summarization was only an issue when the model had a high baseline to start with. Whereas, when baseline performance was low, summarization had negligible effect [Table 1]. Also, the degree of summarization (either syntactic or semantic) was not correlated with performance degradation (figures attached under Results).

Table 1

: Model baseline performance on GPQA Diamond vs Model performance
when the questions were summarized. Results shown with 95% confidence intervals.

Methodology

First, I evaluated GPT 4o-mini, and o3-mini on GPQA Diamond to measure baseline scores. Then, I summarized the questions using GPT 4o, before evaluating the above models on the shortened questions. I didn’t summarize the answer choices.

Summarization

Syntactic Summarization: This was measured as ratio of words (separated on whitespace) between the summarized and the original prompts.[1] This cap...
Reverse Biomimicry 
by cboingo — 2025-04-01T01:19:52.792Z

Biomimicry is a wonderful concept that takes inspiration from nature for the designs of technology. The sleek design of the nose of high speed trains reduces wind-resistance, allowing for more efficient movement of trains that is directly inspired by the way birds’ beaks reduce wind resistance. Ventilation systems take direct inspiration from the ventilation towers in termite mounds. One of my personal favorites involves the use of small strands of wood with seeds attached to them - the wood strip begins to twist in a corkscrew motion as water evaporates from it, eventually twisting the attached seed into the ground - an ingenious method for getting scattered seeds to bury themselves that is directly inspired by what geranium seeds have already evolved the ability to do.

However, I have begun to recognize a phenomenon, which I describe as “reverse biomimicry” that is all too pervasive, particularly within the field of biology, in which we take our own technological developments and use them to describe and understand various biological phenomena. For example, every biology student is taught that DNA is the “code” of life, or that each cell is like a “factory”, or that the human brain is like an extremely advanced “computer”. Code, factories, and computers are all human constructs. DNA is, however, not a code; it’s a sequence of nucleotides. Calling it “code” imposes the metaphor of programming and intention, where in reality it’s biochemical interaction filtered through evolutionary pressure. Likewise, cells are not factories; factories have blueprints and managers. Cells are stochastic, decentralized, emergent systems. And lastly, brains are not computers; they embody experience through plastic, distributed, and context-dependent changes. They are not even like computers—they invented computers.

These metaphors may be useful as teaching tools, but I fear that they result in a drastically over-simplified, and potentially even incorrect conceptualization of some of the natural phenomena that require correct understanding for the most crucial elements of our progress as a species. These conceptualizations are models, and while the adage goes that “the model isn’t correct, but it works”, I fear that this breeds a form of intellectual laziness. We have not made the enormous strides that I felt were implicated when we sequenced the enti...
AI Politics: Polarization and Chaos
by American Psychohistory — 2025-03-31T23:46:38.303Z

Disclaimer: Most futurology turns out to be totally wrong and utterly without foundation. This goes double for unprecedented world-changing tech like AI. Still, it’s difficult for me to imagine that we won’t see some version of the following scenario.

Major Assumption: No Superintelligence Speedrun

We might quickly hit AGI and then use a million AGIs to invent superintelligence in a month. In that case, unless you work at the AI company concerned or are an important government regulator, you will have next to no impact on how things turn out.

For everyone else, it's more important to concentrate on medium/slow takeoff scenarios in which AI is integrated into and disrupts industries and everyday life. This will give the general public, politicians, and activists time to incorporate AI into their agendas.

Polarization: AI as a New Political Battlefield

The economic winners (capitalists and blue collar workers) and losers (educated knowledge workers) during medium/slow AI takeoff already form the opposing sides in the current political war. AI will slot seamlessly into the existing left-right divide, massively exacerbating the culture war by threatening the left's economic livelihood. Chaos, mass protest, and even potential civil war will result.

The Left: Resistance and Radicalization

The liberal side of the spectrum includes knowledge workers, professionals, and students—the people whose jobs will be automated by AI.

The threat of automation will deepen liberals' existing antipathy towards Silicon Valley and big tech. Professionals will form unions and strike to protect their jobs and wages. Meanwhile, college students, facing massive debt for degrees AI will render obsolete by the time they graduate, will join protests. Historically, students have been leaders in protest movements, and now—with AI taking care of homework—they’ll have even more time to mobilize politically.

While personal economic insecurity may be the catalyst, protesters will integrate anti-AI activism into the broader liberal agenda, highlighting AI’s potential for discrimination, the exploitation of internet commons by corporations, environmental impacts, and troubling applications of algorithmic judgment in human lives.

Their demands will likely include a universal basic income, extensive regulatory oversight of AI, mandatory human supervision, and protection of certain jobs as strictly human doma...
Call for Collaboration: Renormalization for AI safety 
by Lauren Greenspan — 2025-03-31T21:01:56.500Z

We invite proposals that probe aspects of

renormalization

in AI systems that will help us predict, explain, and interpret neural network behavior at different levels of abstraction. We import ‘renormalization’ from physics, as a technique to coarse-grain theoretical descriptions of complex interactions to focus on those that are most relevant for describing physical reality. We view this direction as a vast ‘opportunity space’ with many possible points of entry, and have identified a few research programmes as preliminary ‘ways in’. A detailed roadmap of this space, and discussion of the programmes can be found here and here. Our goal is to narrow the theory-practice gap by grounding an abstract analogy into a practical framework capable of directly impacting real-world interpretability and informing

better scientific foundations

for AI safety. A QFT framework for AI systems could give us a toolkit for finding principled features, modeling their interactions at different levels of granularity in their interpretation, and ensure a well-grounded separation between ‘safe’ and ‘unsafe’ behaviors in AI systems. We invite proposals to keep these differences in mind and be clear on which methods or analogies are immediately useful, and which require new development.

We hedge that progress depends on clarifying the link between implicit renormalization – which models how networks coarse-grain information into representations by organizing data into network-meaningful structures – and explicit renormalization, which we operationalize as an interpretability tool capable of probing that structure at a scale of granularity that is meaningful to us. While there is a growing community studying how neural networks implicitly renormalize (e.g.,

Roberts

,

Berman

,

Erbin

,

Halverson

) to organize information (i.e. from data into features) during training and inference, we stress that these are both important, and likely related, even if there is some fuzziness in defining a scale of ‘human interpretation’ and relating it to a network’s implicit notion of scale. Shedding light on this relationship, and leveraging our insights to perform explicit renormalization over neural network representations, is a core goal of this call.

About this Call

At PIBBSS, our goal is to improve our scientific un...
A response to OpenAI’s “How we think about safety and alignment”
by Harlan — 2025-03-31T20:58:31.901Z
Link: https://intelligence.org/2025/03/31/a-response-to-openais-how-we-think-about-safety-and-alignment/

This is part of the MIRI Single Author Series. Pieces in this series represent the beliefs and opinions of their named authors, and do not claim to speak for all of MIRI.

OpenAI recently added a new webpage to share the company’s views on AI safety and alignment. I appreciate that they made this; the companies racing to build superintelligence should be much more active participants in the public debate about whether it will kill everyone. I also appreciate that the page ends by calling for feedback from those who disagree with them.

I have disagreements not only with their strategy, but also with how they communicate that strategy. I find these two types of disagreement difficult to disentangle. There are three main things that I want to say:

The page’s messaging about the expected pace of AGI development is confusing, and I think OpenAI should be more clear about their beliefs.
The page’s discussion about alignment overlooks important considerations that OpenAI has explicitly acknowledged in the past, and I think they should address these considerations more directly.
OpenAI’s stated plan to prevent superintelligence from killing everyone relies on unproven assumptions, and I think they should take responsibility for proving those assumptions.

1. AI timelines doublespeak

In the first section, they say that they changed their mind about the likely pace of AGI development. They now expect it to be a “continuous” process rather than a “discontinuous” moment. But what does that mean?

In AI forecasting discussions, “continuous” is typically used to mean that future progress will be consistent with the trendline you would extrapolate from past data.

In AI forecasting, progress is continuous if a new observation is consistent with the trend you would have extrapolated from past data. (AI Impacts)

Could this be what OpenAI means by “continuous?” Maybe, but it would be a bit odd in context. According to the second section, they expect AI, within a few years, to begin transforming the world to be “more different from today’s world than today’s is from the 1500s.” I think this is actually a reasonable prediction, but it relies on new processes like recursive self-improvement, not simple extrapolation of existing trendlines.

Moreover, according to OpenAI, being in the “continuous” world is a reason to be more optimistic about their iterative, trial-and-error approach to al...
Opportunity Space: Renormalization for AI Safety 
by Lauren Greenspan — 2025-03-31T20:55:52.155Z

This opportunity space was developed with Dmitry Vaintrob and Lucas Teixeira as part of PIBBSS' horizon scanning initiative. A detailed roadmap of can be found

here

.

Background

The basic premise is to view different aspects of a neural network (NN) as a complex statistical system, as follows:

The data-generating process

A NN depends on empirically obtained labeled datasets drawn from some "ground truth" distribution about which the NN must learn to make predictions. By modeling the data-generating process as a physical system (e.g., a Gaussian field theory), we can use renormalization to reason about the ontology of the dataset independently from the NN representation.

NN activations

We can view a NN itself as a statistical system, which transforms an input to an output via a series of intermediate activations. While a trained neural net is deterministic, interpretability seeks suitably “simple”, “sparse”, or otherwise ‘interpretable’ explanations, by coarse-graining the full information about the NN’s activation into a smaller number of “nicer” summary variables, or features. This process loses information, sacrificing determinism for ‘nice’ statistical properties. We can now try to implement renormalization by introducing notions of locality and coarse-graining on the neural net inputs and activations and looking for relevant features via an associated RG flow. While interpreting a trained model via renormalization is an explicit process, we note that this is done implicitly in diffusion models (e.g.

Wang et al., 2023

), which sequentially extract coarse-to-fine features at various noise scales.

The learning process

Because it is randomly initialized in the weight landscape, learning is an inherently stochastic process controlled by an architecture, which outputs a function of an input data distribution implemented by weights.

Of all the processes discussed here, this one most looks like a “physical” statistical theory; in certain limits of simple systems (

Lee et al., 2023

), this process is very well-described by either a free (vacuum) statistical field theory or a computationally tractable perturbative theory controlled by Feynman diagram expansions. Though these approximations fail in more realistic, general cases (

Lippl & Stachenfeld

,

Perin & Deny

), we nevertheless expect them to hold&nb...
Renormalization Roadmap
by Lauren Greenspan — 2025-03-31T20:34:16.352Z

At PIBBSS, we’ve been thinking about how renormalization can be developed into a rich framework for AI interpretability. This document serves as a roadmap for this research agenda – which we are calling an Opportunity Space[1] for the AI safety community. In what follows, we explore the technical and philosophical significance of renormalization for physics and AI safety, problem areas in which it could be most useful, and some interesting existing directions – mainly from physics – that we are excited to place in direct contact with AI safety. This roadmap will also provide context for our forthcoming Call for Collaborations, during which we will hire affiliates to work on projects in this area.

Acknowledgements: While Lauren did the writing, this opportunity space was developed with the PIBBSS horizon scanning team, Dmitry Vaintrob and Lucas Teixeira.

Motivation and Context

In physics, renormalization is used to coarse-grain theoretical descriptions of complex interactions to focus on those that are most relevant for describing physical reality. Here, ‘reality’ is tied to the scale of interest; just as you don’t need to take quantum effects into account to design a safe bridge, the physical descriptions of the same system are different (in a way: emergent) when viewed close up versus far away. To put it differently, renormalization plays two main roles:

To organize a system into “effective” theories at different scales, and
To decouple physical systems into a hierarchy of theories of local interactions by systematically finding so-called “relevant” parameters as you vary the interaction scale. The running of parameters along this scale defines a so-called “RG flow”, which dynamically transforms systems in a way that throws away fine-grained details while preserving coarse-grained behavior.

Like field theories in physics, NNs are highly complex systems with many interacting components. There is evidence that they organize information to learn, for example, ‘features’ that represent the data to varying levels of granularity. This suggests that they’re amenable to renormalization techniques: at a given ‘scale’, irrelevant information can be coarse-grained to appropriately describe a system of interest. These descriptions are linked (or separated) in scale, where each description is hierarchically emergent a...