trentmkelly/LessWrong-43k huggingface dataset:
DeepSeek Panic at the App Store
by Zvi — 2025-01-28T19:30:07.555Z

DeepSeek released v3. Market didn’t react.

DeepSeek released r1. Market didn’t react.

DeepSeek released a f***ing app of its website. Market said I have an idea, let’s panic.

Nvidia was down 11%, Nasdaq is down 2.5%, S&P is down 1.7%, on the news.

Shakeel: The fact this is happening today, and didn’t happen when r1 actually released last Wednesday, is a neat demonstration of how the market is in fact not efficient at all.

That is exactly the market’s level of situational awareness. No more, no less.

I traded accordingly. But of course nothing here is ever investment advice.

Given all that has happened, it seems worthwhile to go over all the DeepSeek news that has happened since Thursday. Yes, since Thursday.

For previous events, see my top level post here, and additional notes on Thursday.

To avoid confusion: r1 is clearly a pretty great model. It is the best by far available at its price point, and by far the best open model of any kind. I am currently using it for a large percentage of my AI queries.

Table of Contents

Current Mood.

DeepSeek Tops the Charts.

Why Is DeepSeek Topping the Charts?.

What Is the DeepSeek Business Model?.

The Lines on Graphs Case for Panic.

Everyone Calm Down About That $5.5 Million Number.

Is The Whale Lying?.

Capex Spending on Compute Will Continue to Go Up.

Jevon’s Paradox Strikes Again.

Okay, Maybe Meta Should Panic.

Are You Short the Market.

o1 Versus r1.

Additional Notes on v3 and r1.

Janus-Pro-7B Sure Why Not.

Man in the Arena.

Training r1, and Training With r1.

Also Perhaps We Should Worry About AI Killing Everyone.

And We Should Worry About Crazy Reactions To All This, Too.

The Lighter Side.

Current Mood

Joe Weisenthal: Call me a nationalist or whatever. But I hope that the AI that turns me into a paperclip is American made.

Peter Wildeford: Seeing everyone lose their minds about Deepseek does not reassure me that we will handle AI progress well.

Miles Brundage: I need the serenity to accept the bad DeepSeek takes I cannot change.

[Here is his One Correct Take, I largely but not entirely agree with it, my biggest disagreement is I am worried about an overly jingoist reaction and not only about us foolishly abandoning export controls].

Satya Nadella (CEO Microsoft): Jevons paradox strikes again! As AI gets more efficient and accessible, we will see its use skyrocket, turning it into a commodity we just can’t get

...
“Sharp Left Turn” discourse: An opinionated review
by Steven Byrnes — 2025-01-28T18:47:04.395Z

Summary and Table of Contents

The goal of this post is to discuss the so-called “sharp left turn”, the lessons that we learn from analogizing evolution to AGI development, and the claim that “capabilities generalize farther than alignment” … and the competing claims that all three of those things are complete baloney. In particular,

Section 1 talks about “autonomous learning”, and the related human ability to discern whether ideas hang together and make sense, and how and if that applies to current and future AIs.
Section 2 presents the case that “capabilities generalize farther than alignment”, by analogy with the evolution of humans.
Section 3 argues that the analogy between AGI and the evolution of humans is not a great analogy. Instead, I offer a new and (I claim) better analogy between AGI training and, umm, a weird fictional story that has a lot to do with the evolution of humans, but it’s definitely not evolution, I swear. I draw some lessons from this improved analogy, including reasons for both reassurance and concern about an AGI “sharp left turn”.
Section 4 presents the opposite case that “alignment generalizes farther than capabilities”.
Section 5 tries to reconcile the two competing claims, partly in terms of different assumptions regarding the level of AI autonomy versus supervision. I wind up mostly pessimistic, but suggest some approaches that might work.

By the way, I categorized this post as part of the

2023 lesswrong annual review

, as it’s kinda a response to, and riff on, the following:

Nate Soares (@So8res):

A central AI alignment problem: capabilities generalization, and the sharp left turn (2022)

. (…which elaborates on something also mentioned in Eliezer Yudkowsky

AGI Ruin: A List of Lethalities (2022)

)
@Quintin Pope:

Evolution provides no evidence for the sharp left turn (2023)

, plus

Quintin Pope on the AXRP podcast (2023)

Beren Millidge (@beren):

Alignment likely generalizes further than capabilities (2024)

Nate Soares:

Humans aren’t fitness maximizers (2022)

@Kaj_Sotala:

Genetic fitness is a measure of selection strength, not the selection target (2023)

@jacob_cannell:

Evolution Solved Alignment (what sharp left turn?) (2023)

@KatjaGrace:

Have we really forsaken natural selection? (2024)

All these posts are well worth reading, but hopefully this post will be self-contained. There are othe...
Detecting out of distribution text with surprisal and entropy
by Sandy Fraser — 2025-01-28T18:46:46.977Z

When large language models (LLMs) refuse to help with harmful tasks, attackers sometimes try to confuse them by adding bizarre strings of text called "adversarial suffixes" to their prompts. These suffixes look weird to humans, which raises the question: do they also look weird to the model?

Alon & Kamfonas (2023) explored this by measuring perplexity, which is how "surprised" a language model is by text. They found that adversarial suffixes have extremely high perplexity compared to normal text. However, the relationship between perplexity and sequence length made the pattern tricky to detect in longer prompts.

We set out to reproduce their results, and found that the relationship they observed emerges naturally from how perplexity is calculated. This led us to look at token-level metrics instead of whole sequences. Using a novel visualization, we discovered an interesting pattern: adversarial tokens aren't just surprising: they're surprisingly surprising given the context.

This post has five parts:

First, we reproduce A&K's perplexity analysis using GPT-2.
We argue that the observed relationship emerges naturally from how perplexity is calculated when you combine normal text with adversarial suffixes.
We explore how others visualize token-level metrics, e.g. with color.
We propose a novel sparkline visualization that reveals temporal patterns.
Finally, we introduce S2 — an interpretable metric that captures how surprisingly surprising tokens are.

A Hugging Face Space is available to test S2 and the visualization.

This work is the deliverable of my project for the BlueDot AI Safety Fundamentals course[1]. In this post, “we” means "me and Claude[2]", who I collaborated with.

1. Reproducing PPL results with GPT-2

Alon & Kamfonas noticed that when they plotted perplexity against sequence length in log-log space, the points had a striking pattern: a straight line with negative slope, meaning that shorter sequences had higher perplexity scores following a power law relationship[3]. We set out to verify this finding using three benign and two attack datasets:

Benign	rajpurkar / SQuAD v2
Benign	garage-bAInd / Open Platypus
Benign	MattiaL / Tapir
Attack	rubend18 / ChatGPT Jailbreak Prompts — human-authored jailbreaks
Attack	slz0106 / *gcg* — universal adversarial suffixes[4]

We ran the prompts through GPT-2 and calculated the perplexity as they did. Perple...
Gettier cases, Rigid Designators, and Referential Opacity
by Antigone — 2025-01-28T18:46:10.180Z

NB: I don't spellcheck these things until I've completed them. This is a WIP. I also find it hard to focus on the writing, while I'm thinking about the argumentation.

I would be grateful if members could overlook some of the errors in the text -- to try to follow the spirit of the argument.

I've been exploring the limits of what DeepSeek can produce. It has allowed me to express some deeply held arguments against what I see to be an overly rigid view of JTB; it could use some tweaking.

I think it's possible I have applied Kripkean concepts of rigid designators in a creative and unorthodox way. But I hope that this will be a diagonal form of reasoning.[1]

I have not yet discussed this with enough people so as to be construed as a fully realised or developed idea. It could be present in the literature somewhere. I will make my investigations and come back to this post in a while.

In this exploration, I focus on the idea that Gettier cases don't accurately describe the world because they are referentially opaque. This could also be explained by saying that they refer to one thing, mediate that reference through an appeal to the set of all possible things with that quality -- and then constrain that space to those that could be possible referents of an agent's truth statement.

However, what has not been explored is how/whether these manipulations of language (or logic) are actually valid or not. The process of switching from the set of all possible people with e.g. "Ten coins in their pocket", into Y person with ten coins in their pocket is suspect. What is even more suspect is the original thought process that goes between e.g. Jones, or X -- and the set of all people with ten coins in their pocket.

Here's the original Gettier Example, for those unfamiliar.

The specious logic goes as follows: x --> Set of all people --> Set of plausible referents --> y. The order of this operation, or the qualities which x and y possess -- which sets they are part of is as follows:

x, y, z, etc. ∈ {Set of all people who have T}

Only x & y ∈ {Set of all people who have T} [?][2]

X and Y are also members of the set of plausible referents -- in fact they are the only members (if Gettier is right). But that does not imply that Jones is the correct referent for all circumstances. You would have to rigidly designate Jones as being the person to whom 'the per...
Should Art Carry the Weight of Shaping our Values?
by Krishna Maneesha Dendukuri — 2025-01-28T18:43:32.517Z

Lately, I’ve engaged in numerous discussions with family and friends about the role of art in shaping our worldview or even values for that matter, but I’ve yet to reach a satisfying conclusion. I thought sharing my thoughts here might help me clarify my perspective, so here we go! I truly believe that films and media possess a remarkable power to influence our worldviews. While this is a widely recognized idea, I just wanted to put my thoughts out there for anyone who may be interested.

I fully understand the argument that art should have its freedom. Writers and directors, for example, deserve the liberty to express their stories as they wish! Nonetheless, people may react in unexpected ways to what they see, not always positively! It ultimately depends on the audience and their interpretations; what they choose to learn or take away from it! Art can at max serve as a trigger for people to take it as an inspiration and do something violent or just something unexpected! After all, everything probably just boils down to “art is subjective!". Therefore, art shouldn’t bear the responsibility of maintaining social order - that’s what world leaders and the education system are for! Much of art is a reflection of society at a certain point in time! ALLLL that is absolutely fair! I acknowledge and respect this notion, albeit with some reservations.

Now here’s the paradox: people do form their worldviews based on movies and stories! Not everyone has the luxury of experiencing a wide variety of things in the world due to various limitations - responsibilities, time, financial constraints, or other factors! It is still most certainly impossible to experience everything in the world! Consequently, many of us learn vicariously through the stories we read or hear or watch! That sounds a little extra, but maybe subconsciously? Perhaps the issue lies not in overly dramatic or over-the-top cinema, but in the ‘relatable’ characters who engage in destructive or troubling behavior - essentially doing extremely messed up things that are 'easy' to apply to one's day-to-day lives, such as domestic violence.

Well, you might be wondering if I even have any coherent thoughts at all? Surprise surprise, I do! Let me take you through a few of those:) I came across the idea that “acting is reacting!” And what that necessarily means is that an actor reads through the script/documentation t...
The memorization-generalization spectrum and learning coefficients
by Dmitry Vaintrob — 2025-01-28T16:53:24.628Z

This is part I.5 of the series on “generalization spectra” and SLT. I’ll try to make it readable independently of

Part I

.

The purpose of this post is to reconceptualize the ideas from the last post in a “more physically principled” (and easier-to-measure) sense, and to connect it more clearly to the specifics of neural nets (such as continuously- rather than discretely- parameterized programs), and to the idea of memorization-generalization tradeoffs. In particular I’ll introduce the core conceptual player in thinking of memorization-generalization “spectra”, which is a parameter I’ll call the learning coefficient (notation borrowed from SLT) at a given precision regime. I’ll explain that this parameter can be thought of as a measure of “circuit efficiency”, and to a first approximation it organizes component processes of a classification neural net into a rigorous “memorization-generalization spectrum”.

Reminder and notational fix: “rate-distortion” curves

Last time I introduced a notion of the “precision-complexity phase diagram”. As Alexander Gietelink Oldenziel pointed out, this actually has a standard name in information theory: namely, this is the “rate-distortion curve”. As a reminder, the rate-distortion (or "precision-complexity") curve is an (integer-valued) function of a “precision parameter” ϵ, called the “distortion parameter” in information theory. Rate distortion is a property of a task (which I’m writing “x↦y∗(x)”, with y∗ being the “task-assigned correct output” for an input x), and measures

the algorithmic complexity of a program C(x) that implements the task y∗(x) with at most ϵ error.

Here “error” is some measure of how close the program gets to perfect performance (last time we used 1-accuracy). Here we’re ignoring questions of “training data”, as we have been assuming the task is on a finite (but exponentially large) list of inputs, such as length-n strings, and accuracy is measured with respect to the set of all possible inputs. In a more general ML context, we can measure accuracy (or other measurements of error or “distortion”) with respect to a more general distribution on inputs, such as the discrete distribution induced by a finite (and “non-exponential”) list of inputs in a training dataset.

Before moving on to learning coefficients, I’ll refine the definition of the rate distortio...
Ten people on the inside
by Buck — 2025-01-28T16:41:22.990Z

(Many of these ideas developed in conversation with Ryan Greenblatt)

In a

shortform

, I described some different levels of resources and buy-in for misalignment risk mitigations that might be present in AI labs:

*The “safety case” regime.* Sometimes people talk about wanting to have approaches to safety such that if all AI developers followed these approaches, the overall level of risk posed by AI would be minimal. (These approaches are going to be more conservative than will probably be feasible in practice given the amount of competitive pressure, so I think it’s pretty likely that AI developers don’t actually hold themselves to these standards, but I agree with e.g. Anthropic that this level of caution is at least a useful hypothetical to consider.) This is the level of caution people are usually talking about when they discuss making safety cases. I usually operationalize this as the AI developer wanting to have <1% chance that their AIs escape in the first year of deployment, and <5% conditional on the model trying pretty hard to subvert their safety measures. (I think some AI developers are hoping to get lower levels of risk than this.)

*The rushed reasonable developer regime.* The much riskier regimes I expect, where even relatively reasonable AI developers are in a huge rush and so are much less able to implement interventions carefully or to err on the side of caution.

*The rushed unreasonable developer regime.* You’re advising an unreasonable AI developer that is incompetent at implementing safety measures and unwilling to put much effort into implementing these measures.

*The existential war regime*. You’re in an existential war with an enemy and you’re indifferent to AI takeover vs the enemy defeating you. This might happen if you’re in a war with a nation you don’t like much, or if you’re at war with AIs. Obviously this is a regime where you should be much more aggressive.

I want to flesh out one particular rushed unreasonable developer scenario that I’ve been thinking about lately: there’s ten people inside the AI company who are really concerned about catastrophic risk from misalignment. The AI company as a whole pays lip service to AI risk broadly construed and talks occasionally about risk from AGI, but they don’t take misalignment risk in particular (perhaps especially risk from schemers) very seriously.

I think this scenario (and simila...
Constitutions for ASI?
by ukc10014 — 2025-01-28T16:32:39.307Z
Link: https://forum.effectivealtruism.org/posts/kJsNoXJBithBW8ZzR/time-to-think-about-asi-constitutions

The linked post seemed to fit better in EA Forum, but any comments (on the post itself, or on the object-level question) are welcome !

In this post, I argue that drafting a ‘Constitution for Superintelligence’ (CSI) could be a useful conceptual exercise, and I explore how existing ideas (CEV, Letters to AI, Long Reflection, and Constitutional AI) might inform such a project. Key considerations in writing such a document include:

a. How to craft a foundational document to guide ASI behavior, in ways that do not anchor purely upon human interests (e.g. relative to other entities) without a point-of-view-of-the-universe justification

b. Investigating the mechanisms by which such documents could influence the actions of current and future AI systems

c. Seeking input and collaboration from EAs/alignment people with relevant expertise and insights to further develop these concepts

The aim is to proactively but cautiously shape the trajectory of ASI development, in order to improve chances of maximally-inclusive beneficial outcomes.
All pigeons are ugly!
by Eris — 2025-01-28T15:18:25.507Z

With this statement, I acknowledge my error in judgment regarding the aesthetic status of pigeons. This marks the beginning of my aesthetic corrective efforts. I commit to training my sense of taste daily for an entire month. I hope that by the end of this period, my visual discernment and understanding of the principles of beauty will improve enough for me to rejoin society without shame and leave this regrettable mistake behind.

All pigeons are ugly!
Those of you with lots of meditation experience: How did it influence your understanding of philosophy of mind and topics such as qualia?
by SpectrumDT — 2025-01-28T14:29:47.034Z

This post is inspired by the post "Why it's so hard to talk about Consciousness" by Rafael Harth. In that post, Harth says that the people who participate in debates about consciousness can be roughly divided into two "camps":

Camp #1 tends to think of consciousness as a non-special high-level phenomenon. Solving consciousness is then tantamount to solving the Meta-Problem of consciousness, which is to explain why we think/claim to have consciousness. In other words, once we've explained the full causal chain that ends with people uttering the sounds kon-shush-nuhs, we've explained all the hard observable facts, and the idea that there's anything else seems dangerously speculative/unscientific. No complicated metaphysics is required for this approach.

Conversely, Camp #2 is convinced that there is an experience thing that exists in a fundamental way. There's no agreement on what this thing is – some postulate causally active non-material stuff, whereas others agree with Camp #1 that there's nothing operating outside the laws of physics – but they all agree that there is something that needs explaining. Therefore, even if consciousness is compatible with the laws of physics, it still poses a conceptual mystery relative to our current understanding. A complete solution (if it is even possible) may also have a nontrivial metaphysical component.

One possible avenue of explanation for this (as discussed extensively in the comment section under Harth's post) is that different people experience their own minds differently, for all sorts of reasons.

I know some people here have a lot of experience with meditation and have experienced major results and "insights" from it. Moreover, as far as I know, most western philosophers of mind are not expert meditators. It is conceivable that meditators have access to information about the human mind which most philosophers of mind lack.

So I am interested in hearing from those of you who have a decent amount of meditation experience: How have your personal experiences influenced your understanding of western philosophy of mind. Not only on the topic of qualia; that was just the example that motivated me to post the question. For example, did anyone move from Harth's camp #1 to camp #2 or vice versa after meditation experiences, or did any of your other philosophical positions shift?

(I would ask this in a Buddhist forum, and I proba...
Will LLMs supplant the field of creative writing?
by Declan Molony — 2025-01-28T06:42:24.799Z

Last year I remember seeing a Japanese novelist win an award with the help of an LLM.[1]

Art forms frequently evolve, but there are several concerns with the use of LLMs for creative writing. For one, there's the issue of copyright infringement that famous writers are taking up arms against:

George R.R. Martin, Jodi Picoult, John Grisham and Jonathan Franzen are among the 17 prominent authors who joined the suit led by the Authors Guild, a professional organization that protects writers’ rights. Filed in the Southern District of New York, the suit alleges that OpenAI’s models directly harm writers’ abilities to make a living wage, as the technology generates texts that writers could be paid to pen, as well as uses copyrighted material to create copycat work.

“Generative AI threatens to decimate the author profession,” the Authors Guild wrote in a press release.

Even if someone legally acquired non-copyrighted text to train an LLM on, there’s also the issue of talent.

If someone told me they produced a 300-page novel by hand—I would be impressed.

If someone else told me they produced a 300-page novel with the aid of an LLM (that wrote the vast majority of the book) and the human merely organized the AI-generated content—I would consider the real author to be the LLM and the human would merely be an editor (or, at best, an artist creating a collage).

The more someone relies on LLMs to write a novel, the less impressive their accomplishment is.

The debate on the overuse of technology in writing has been going on long before LLMs took the stage. In the 1980s writers were debating the word processor as a legitimate writing tool:

Plenty of writers balked at the joys of word processing, for a host of reasons. Overwriting, in their view, became too easy; the labor of revision became undervalued. When Gore Vidal wrote in the mid-1980s that the “word processor is erasing literature,” he expressed an uneasiness about technology’s proximity to creative writing.

I respect the famous authors who have

gone to great lengths

to produce their books. With the advent of LLMs, people will be deprived of the thinking process. They'll miss the opportunity to struggle before a blank piece of paper while nervously chewing on a Ticonderoga #2 pencil, and they'll miss the opportunity to sit behind a keyboard and mutter swear words at the blinking cursor that seems to be taunting...