trentmkelly/LessWrong-43k huggingface dataset:
Text First, Evidence Later? Managing Quality and Trust in an Era of AI-Augmented Research
by Thehumanproject.ai — 2025-04-10T18:52:58.934Z

It's hardly a secret anymore that researchers across disciplines are increasingly turning to Large Language Models (LLMs) like ChatGPT. Initially adopted perhaps for polishing prose or overcoming writer's block, their use has rapidly evolved. We now see LLMs employed not just for superficial text enhancement, but for brainstorming, structuring arguments, summarizing complex papers, and even generating draft sections. This integration marks a significant shift in the academic workflow, presenting both possibilities for efficiency and profound challenges to the integrity of the research process itself. The implications are far-reaching, forcing those of us working in academia and research to confront uncomfortable questions about authorship, oversight, and the very nature of scholarly contribution in the age of AI.

This shift was driven home for me during a recent alumni meet-up. Colleagues from various fields shared anecdotes painting a picture that could be described as a dark reality of current academic practice. They described a cycle where LLMs are used to draft manuscripts, then LLMs are used by the peer-reviewers to critique them, followed by the original authors using LLMs again to address the feedback. This iterative process continues, seemingly until both the human reviewer (perhaps cursorily) and the respective AI systems are satisfied. One might argue this streamlines the notoriously burdensome peer-review process, potentially standardizing feedback or making it easier for non-native English speakers to navigate the publication process. However, the underlying concern is the potential evaporation of deep human thought, critical engagement, and genuine intellectual oversight in this AI-mediated loop.

Frankly, this AI-driven acceleration is happening atop a system already showing deep cracks. The traditional peer review model is arguably broken, or at least severely strained. Reviewers, the gatekeepers of quality, typically receive no compensation for their time and expertise. Simultaneously, authors often face significant Article Processing Charges (APCs) to get their work published. Compounding this is the relentless "publish or perish" culture within academia, incentivizing quantity often at the expense of quality. Now, add the ease with which AI can generate plausible-sounding text: the sheer volume of submissions, including poten...
Nuanced Models for the Influence of Information
by ozziegooen — 2025-04-10T18:28:34.082Z

A common fallacy I see regarding information sources is the assumption that they either have zero impact or completely determine critical decisions.

"Our World In Data isn't directly cited by the US President, therefore it's worthless."
"Prediction markets haven't been adopted by top elected officials, so they contribute zero value."
"Prediction markets are already more competent than many intellectuals, and people don't listen to them. Therefore, there's no value in them being more competent. It's a bottleneck issue, and all that matters now is marketing."
"LLMs can't match the reasoning of elites, therefore they offer nothing."
"The moment LLM forecasters outperform human forecasting, in certain experiments, LLMs will completely take over."

This binary thinking misrepresents how information actually propagates through systems. In reality, valuable insights rarely travel directly to ultimate decision-makers. The Hollywood model (i.e. the highly simplified model) of direct access to power oversimplifies real-world influence dynamics.

Instead, impact diffuses gradually through networks. An article might influence Sally, who shares the concept with Mark, who develops it further before passing it to Will. Information sources create ripple effects with varying degrees of influence throughout human networks - strongest at the source but still propagating outward.

This doesn't mean "systems are complex, therefore all information is equally valuable." We can still confidently predict that certain content will gain no traction. I'm very supportive of work to better model the impacts of information, but suspect that it would take some nuance.

Human influence (i.e. Power) operates similarly. Individuals exert varying degrees of impact on different nodes within large systems. Their immediate sphere of influence matters, as does the proximity of those connections to other significant actors.

Lastly, I'll recommend the metaphor of "a voice in the room." Adding another thinker to a decision-making group creates a limited but non-zero contribution. They'll speak on topics where they add value and remain silent elsewhere, producing neither zero nor total influence.

For example, I suspect that LLMs are clearly "a voice in the room" of limited impact in many key situations now, and I expect that it will gradually become a louder and louder voice in the room.

I'm unsure how best way...
Playing in the Creek
by Hastings — 2025-04-10T17:39:28.883Z
Link: https://hgreer.com/PlayingInTheCreek

When I was a really small kid, one of my favorite activities was to try and dam up the creek in my backyard. I would carefully move rocks into high walls, pile up leaves, or try patching the holes with sand. The goal was just to see how high I could get the lake, knowing that if I plugged every hole, eventually the water would always rise and defeat my efforts. Beaver behaviour.

One day, I had the realization that there was a simpler approach. I could just go get a big 5 foot long shovel, and instead of intricately locking together rocks and leaves and sticks, I could collapse the sides of the riverbank down and really build a proper big dam. I went to ask my dad for the shovel to try this out, and he told me, very heavily paraphrasing, 'Congratulations. You've cracked the river damming puzzle. And unfortunately, this means you no longer can try as hard as you can to dam up the creek." The price of victory is that the space of games I could play was permanently smaller, and checkmate was not played on the board. This is what growing up looks like.

I experienced many variations. After I put a hole in the ceiling, building K'Nex catapults switched from a careless activity to a careful activity. On the more supervised side, when I was curious how big of an explosion I could make by taping together sparklers, I was not allowed to find the limit. After I cracked the trick of tillering, I could no longer try as hard as I could to build bows and arrows for the backyard battles my friends and I engaged in. I was and am particularly proud of earning this ban.

A couple years back, I got a job offer from an investment bank to help them win zero sum games against people who didn't necessarily deserve to lose. I had tried very hard to get that offer: leetcoding, studying build systems, crafting my resume. It was only once I got it that I realized I no longer could play the game "make as much money as I can." I had found the equivalent of smashing the river banks with a shovel and needed a new game.

There's an enormous amount of nuance here. If I find something that's sufficiently stronger than me, I can play hardball for a long time. My favorite game at the beach is playing in the outflow of a tidal pool at low tide. The river is pouring across the sand, but I, a child again, am stronger than river. My hands can dam it, make lakes, deltas, and canyons. The incoming tide is s...
The Three Boxes: A Simple Model for Spreading Ideas
by JohnGreer — 2025-04-10T17:15:55.163Z

This is cross-posted from my blog.

We need more people on board for life extension in order to hit longevity escape velocity in our lifetimes. But most people have never heard of life extension, and even those who have often follow the same predictable arguments. “What if it doesn't work?” “What if bad people live forever?” “What if humanity needs to refresh its stock every so often in order to make progress?” “What about heaven?”

We need more people working on AI safety so we don't all end up dead.

We need more people to understand the coordination problem as the central problem in politics and economics.

For those of us in the business of spreading ideas, it can be a tough row to hoe (I had to look up this phrase to make sure I was saying it right). It can feel like an uphill battle but the way you model it can help guide you.

This is my model when thinking about spreading ideas, especially ones outside the Overton window. It applies less in the later stages of a movement when the Overton window has shifted.

It’s about three boxes. I like buckets but I already have an important post about three buckets so boxes it is.

There’s the giant box. This box has the people that don’t agree with you. In fact, they probably think you’re a terrible person just for talking about these ideas.

There’s the small box. This box has the people that already agree with you.

Then there’s the Tiffany-sized box. These are the people who you may be able to reach with your ideas. The box is much smaller than the giant box.

Why is this?

Four reasons:

1. Efficient market hypothesis. The people who are open to these ideas already believe them.

The obvious caveat is the EMH applies less to new or radical or uncommon ideas since people are less likely to have heard about them. Think transhumanism vs democracy or AI safety research vs Christianity.

2. The amount of people who are smart enough/high enough in openness/reasonable enough to understand and accept a new idea is small.

3. More often than not, the truth is something we don’t want to hear or don’t want to be true. I would love if there was an eternal paradise but there’s not. It would be wonderful if everyone was born smart and beautiful but they aren’t. I would love if we knew for certain that cryonics would work but we don’t.

4. Rationality people can be worse at selling ideas than non-rationality people for a number of reasons:

a. Autism...
Reactions to METR task length paper are insane
by Cole Wyeth — 2025-04-10T17:13:36.428Z

Epistemic status: Briefer and more to the point than my model of what is going on with LLMs, but also lower effort. This is my most "combative"-toned post to date and I think a little too much so.

Here is the paper. The main reaction I am talking about is AI 2027, but also basically every lesswrong take on AI 2027. (EDIT: A couple of the authors of AI 2027 have pointed out that its predictions are not mainly based on METR's report, but I think the main points of this post still hold up).

A lot of people believe in very short AI timelines, say <2030. They want to justify this with some type of outside view, straight-lines-on-graphs argument, which is pretty much all we've got because nobody has a good inside view on deep learning (change my mind).

The outside view, insofar is that is a well-defined thing, does not justify very short timelines.

If AGI were arriving in 2030, the outside view says interest rates would be very high (I'm not particularly knowledgeable about this and might have the details wrong but see the analysis here, I believe the situation is still similar), and less confidently I think the S&P's value would probably be measured in lightcone percentage points (?). We might also look at the track record of previous (wrong) predictions that AGI was near dating back as far as the 1960's or so. Also, we might use the predictions of experts and superforcasters. Only the last (and perhaps least pure) "outside view" seems to support short timelines at all, but depending on the way the question is posed it still seems to predict more like 2050.

As far as I can tell, this is why everyone keeps using this plot instead of those other methods:

I'm glad METR did this work, and I think their approach is sane and we should keep adding data points to this plot. Also, over the last couple of years this has been one of my only updates towards shorter timelines - almost everything else has pushed me towards longer timelines (I don't expect to be able to change anyone's intuitions here, but I look outside and do not see AGI reshaping the world, do not find chatbots particularly useful, etc. and it's clear that this is not a central prediction of my short-timelines-conditioned model).

The plot doesn't say what people want it to say. There are three major problems.

I'm not sure which is most serious, but I think it's probably tha...
Existing Safety Frameworks Imply Unreasonable Confidence
by Joe Rogero — 2025-04-10T16:31:50.240Z
Link: https://intelligence.org/2025/04/09/existing-safety-frameworks-imply-unreasonable-confidence/

This is part of the MIRI Single Author Series. Pieces in this series represent the beliefs and opinions of their named authors, and do not claim to speak for all of MIRI.

Most human endeavors have bounded results. A construction project may result in a functional bridge or a deadly collapse, but even catastrophic failure will not kill a billion people. Both success and failure are bounded, and the humans undertaking such a project can make reasonably correct estimates of those bounds.

The development of frontier artificial intelligence often thwarts any concrete expectation. Leading developers talk of scenarios as extreme as human extinction from AI, but there is heavy disagreement and uncertainty about when particular disaster scenarios are likely to arise. You can guess at a model’s properties ahead of time, but this is always a guess, no matter how informed. Models’ behavior is increasingly goal-oriented and difficult to predict. So when doing frontier development, you don’t know the bounds you’re working in.

You can run thought experiments, or get all of the smartest people you know together, and have them tell compelling stories about plausible dangers. You can compile these worrying scenarios into a list and distill from that list a compendium of plausible early warning signs for each scenario, and you’ll have a loose account of situations and behaviors to monitor for.

Importantly, though, such plans would be incomplete in potentially fatal ways. The list generated by such a process is not comprehensive. The tracking of worrying scenarios is imperfect, limited as it is by available tools and understanding. Further, with frontier AI we should expect not just the unexpected, but the unexpectable—outcomes generated by a profoundly alien system, or a system that surpasses humans in unprecedented ways.

The high-level goals of a risk management framework in this context ought to include:

Estimating the total risk to society posed by a given approach, and
Halting any approach whose estimated risk exceeds expected benefits.

Questions it should be attempting to answer include:

What is the chance we fail to anticipate a potentially dangerous capability?
What is the chance we fail to detect the attainment of a dangerous capability?
What is the chance we cannot provide adequate mitigation?
What is the residual risk after all available mitigations are applied?

The lab...
Arguments for and against gradual change
by Gustavo Ramires — 2025-04-10T14:43:53.489Z

Essentially all solutions in life are conditional: you apply them in the right context, in the right conditions to achieve a good outcome. Obviously banging a hammer on your desk
does probably no good, while banging a hammer on a nail you want to use to hang that nice painting on the wall may be a great idea :)

I want to talk about gradual change, and it doesn't apply all the time. Sometimes if you are certain enough of an outcome and things are urgent gradual change is unnecessary and even harmful.

I think the context in which gradual change is great is when there is significant conflict and controversy. Or when discussions and ideas are turning unstable (violently swinging
between extremes), when there is uncertainty.

I'd say this applies to a lot of current politics and technology (as of 2025). There is talk of radical change, often in opposing directions. This can risk fracturing and destabilizing societies, compromising the basis that has been carefully built upon for decades or centuries.

Here are some closely related arguments for (and against) gradual change:

(1) (For) If a lot of something is good, usually a little of something is good too.

Effects being linear is a pretty reasonable default assumption unless there are reasons for otherwise. And if not linear, usually approximately monotonic holds. Which is to say, usually a little medicine is better than no medicine (in case a large dose is good), a little salt is better than no salt (in the case a lot of salt is good), and so on.

So there's usually no harm, compared to the baseline, of trying a little change. If said change has the desired positive effect, do more of it, repeatedly, until you're doing it enough.

An example of a non-monotonic system to change might be a program. People don't change programs by changing single characters and recompiling. You need a minimum viable change. A too small change (say if you were restricted to only be able to change a few characters of a source code each deployment) is probably going to affect the system negatively (making code more difficult to understand and not fully able to enact a desired change). But usually fairly small (a few lines) changes, which are then tested and possibly deployed, are good from an engineering and reliability standpoint.

(2) (For) The risk of a little of something is much smaller

By a similar argument, if that thing is bad, usually a littl...
Disempowerment spirals as a likely mechanism for existential catastrophe
by Raymond Douglas — 2025-04-10T14:37:58.301Z

When complex systems fail, it is often because they have succumbed to what we call "disempowerment spirals" — self-reinforcing feedback loops where an initial threat progressively undermines the system's capacity to respond, leading to accelerating vulnerability and potential collapse.

Consider a city gradually falling under the control of organized crime. The criminal organization doesn't simply overpower existing institutions through sheer force. Rather, it systematically weakens the city's response mechanisms: intimidating witnesses, corrupting law enforcement, and cultivating a reputation that silences opposition. With each incremental weakening of response capacity, the criminal faction acquires more power to further dismantle resistance, creating a downward spiral that can eventually reach a point of no return.

This basic pattern appears across many different domains and scales:

HIV progressively destroys the immune system designed to fight it.
Anxiety, burnout, or depression deplete executive function, which is required for taking steps to address the problem.
Cults methodically isolate members from support networks that might help them leave.
Corporate toxic cultures drive away the talented employees most capable of fixing them.
Political polarization erodes the trust necessary for collective problem-solving.

In each case, the threat doesn't just cause damage — it undermines the capacity to respond to that very threat.

Abstracting:

A disempowerment spiral is a feedback loop in which an actor faces an ongoing threat that somehow disempowers the actor’s capacity to respond to that threat. As the actor’s response capacity decreases, they become even less able to prevent further disempowerment.

The core disempowerment loop for HIV

In this article, we propose disempowerment spirals as a lens for analysing how complex systems fail. Our primary motivation is to better understand existential risks (including AI risk) — since we cannot directly observe existential catastrophes, we need indirect methods to understand them. Disempowerment spirals in particular provide a possible answer to the question “

if something gets bad enough, why don’t people just stop it?

”.

In the rest of this post, we will first draw out some general observations about disempowerment spirals, and then in the last section turn to a discussion of what this migh...
AI #111: Giving Us Pause
by Zvi — 2025-04-10T14:00:04.194Z

Events in AI don’t stop merely because of a trade war, partially paused or otherwise.

Indeed, the decision to not restrict export of H20 chips to China could end up being one of the most important government actions that happened this week. A lot of people are quite boggled about how America could so totally fumble the ball on this particular front, especially given what else is going on. Thus, I am going to put these issues up top again this week, with the hopes that we don’t have to do that again.

This week’s previous posts covered AI 2027, both the excellent podcast about it and other people’s response, and the release of Llama-4 Scout and Llama-4 Maverick. Both Llama-4 models were deeply disappointing even with my low expectations.

Upgrades continue, as Gemini 2.5 now powers Deep Research, and both it and Grok 3 are available in their respective APIs. Anthropic finally gives us Claude Max, for those who want to buy more capacity.

I currently owe a post about Google DeepMind’s document outlining its Technical AGI Safety and Security approach, and one about Anthropic’s recent interpretability papers.

Table of Contents

The Tariffs And Selling China AI Chips Are How America Loses. Oh no.

Language Models Offer Mundane Utility. They know more than you do.

Language Models Don’t Offer Mundane Utility. When you ask a silly question.

Huh, Upgrades. Gemini 2.5 Pro Deep Research and API, Anthropic Pro.

Choose Your Fighter. Gemini 2.5 continues to impress, and be experimental.

On Your Marks. The ‘handle silly questions’ benchmark we need.

Deepfaketown and Botpocalypse Soon. How did you know it was a scam?

Fun With Image Generation. The evolution of AI self-portraits.

Copyright Confrontation. Opt-out is not good enough for OpenAI and Google.

They Took Our Jobs. Analysis of AI for education, Google AI craters web traffic.

Get Involved. ARIA, and the 2025 IAPS Fellowship.

Introducing. DeepSeek proposes Generative Reward Modeling.

In Other AI News. GPT-5 delayed for o3 and o4-mini.

Show Me the Money. It’s all going to AI. What’s left of it, anyway.

Quiet Speculations. Giving AIs goals and skin in the game will work out, right?

The Quest for Sane Regulations. The race against proliferation.

The Week in Audio. AI 2027, but also the story of a deepfake porn web.

AI 2027. A few additional reactions.

Rhetorical Innovation. What’s interesting is that you need to keep

...