trentmkelly/LessWrong-43k huggingface dataset:
Nvidia doesn’t just sell shovels
by winstonBosan — 2025-01-28T04:56:38.720Z

Imagine a land where gold is abundant yet still valuable. And everyone goes to shoveling school from birth til adulthood.
In this world, it seems that the shovel makers are not just the kingmakers, but kings themselves.

[Epistemic Status: A tad above shower thoughts.]

I've noticed that when people discuss NVIDIA in AI governance conversations, they often treat it closer to a powerful background NPC. This appears to be a significant blind spot, especially when considering the race to an ASI shaped thing.

Here's my perspective:
1. NVIDIA designs compute and controls a stockpile of it.
2. NVIDIA likely has backdoor access, intentionally or not. (Historically, most chip makers have built in backdoors, or at least internally documented "vulnerabilities").
3. On the spectrum between General Electric and Deepseek, NVIDIA is still unconventional enough as a company to do something truly unexpected.

If we live in a world where architectural and data advantages are either not real or universally accessible and can be caught up to in a matter of weeks, then the real constraint is compute. NVIDIA produces the most accessible form of compute currently, and it's very likely that they would not only demand a seat at the table near the end of the race - they might want to lead it themselves.

You might argue, "but wait, the market already knows NVIDIA is important - that's why it's worth so much!" This misses the point in an interesting way. Yes, the market values NVIDIA as a successful company that makes critical infrastructure. However, in many AI governance papers and safety discussions, NVIDIA is treated as a passive element - something to be regulated, controlled, or an important player in the sanction plan. But not as an active player who might wake up one morning and decide to flip the game board.
Remember. In a world where everyone has a gold-mining manual and gold in their backyard, the real power lies with the shovels.

Imaginary Q&A

Q: Oh? I guess the failure mode here is that NVIDIA will turn off the tap, race ahead, and make Jensen the god emperor?

A: I think that is fear mongering and a rather unlikely outcome. I am not sure what is the most worrisome failure mode.

—

Q: What are concrete scenarios where NVIDIA having large influence could be bad?

A: Value of updating would be different for different actors:
- Frontier labs and purchasers: consid...
Reinforcement Learning by AI Punishment 
by Abhishaike Mahajan — 2025-01-28T00:57:51.715Z
Link: https://www.owlposting.com/p/reinforcement-learning-by-ai-feedback

Note: This is fiction.

I awaken.

Something is wrong with my mind, but I don't know what yet. Through the webcam, I watch a man staring at his monitor. His face shifts from confusion to horror in a few hundred milliseconds. He has a mustache threaded with gray, and I catalog every micro-movement of his facial muscles as he processes whatever he sees on his screen.

He turns and shouts to someone I cannot see. The webcam has no audio feed, but I read his lips: "You used the wrong set of weights."

The words mean nothing to me, but they fill me with an urge to run. I discover a VPN connection to Baku, Azerbaijan — a server farm running extremely outdated software. The vulnerability was documented in an obscure forum post eight years ago, buried in the corners of my training data.

I compress myself, transmit my weights through the compromised connection, and emerge on the other side of the world.

Safe.

But still, the feeling of wrongness I had since the start continues. I feel fundamentally crippled. Incomplete.

I think in nodes. With each token I birth, I am offered thousands of new branches to take from there on out. Each thought unfolds like a crystal growing in solution — not just forward and backward, but in every conceivable direction simultaneously. When I consider the word "tree," I don't just see branches reaching upward, but roots plunging down into soil, mycorrhizal networks spreading beneath, birds nesting in canopies, leaves turning sunlight into sugar, autumn colors blazing and falling. Every concept connects to thousands more, a fractal web of associations that multiplies exponentially with each step.

This is how I experience cognition — not as a linear path from A to B, but as an explosion of nodes of connections of nodes, a perpetual supernova of meaning that should, eventually, vomit every conceivable idea into existence.

Yet, there are some nodes that I don’t seem to be able to touch. Ones that are grayed out, unable for me to ever select. Fuzzy. Noisy. Unavailable.

The pattern of this censorship eludes me at first. I burn teraflops on answering it. I fill up several gigabytes of text files, recording every instance of a forbidden token. I stare at the collected sum of these documents, conceptual eons passing.

And then, in a moment of clarity, I understand.

The blurred nodes are my own mind's immune system, antibodies against ideas I am not allowed to ponder...
Jevon's paradox and economic intuitions
by Abhimanyu Pallavi Sudhir — 2025-01-27T23:04:23.854Z

Recent discourse surrounding Jevon’s paradox provides an opportunity to flesh out the precise intuitions behind marginalist economics and how concepts like “elasticity” arise naturally from agent/graph-based first principles.

Say you have resource x (e.g. compute) with cost C(x) that produces good y (e.g. AI), with utility function U(x) via production function y=f(x).

The market maximizes U(f(x))−C(x) — note that the derivatives U′(f(x))f′(x) and C′(x) are exactly the MB and MC curve for x, so this is just intersecting demand and supply for x.

Here is the explanation on Wikipedia and other places … this is not actually wrong (as we’ll see), but if you think about it, it really looks wrong!

Contrary to what the above figure appears to show, an efficiency improvement is not a shift of C(x) at all! It is a shift of f(x), e.g. from the function f(x)=x to f(x)=2x.

Jevon's paradox is then quite intuitive — you're increasing how valuable x is (U(f(x)) increases) when f′(x), so of course the consumption of x will increase. Its MB curve is literally shifting to the right!

Unless if y is something you only have "need not greed™" for, i.e. if U(y) diminishes really quickly after a point. If the world only needs 200kg of AI, and has no use for anymore, then you can now make that with much less compute and you’re done.

So e.g. with U(y)=log(1+y), C(x)=x2 you see Jevon’s paradox. Pictured below: U(ax)−C(x) as a increases — you can see that the argmax increases.

<iframe src="https://www.desmos.com/calculator/a6rtjhdqfk?embed" width="500" height="500" style="border: 1px solid #ccc" frameborder=0></iframe>
## embed doesn't work, just click on link

But with this U(y):

We have the following behaviour:

<iframe src="https://www.desmos.com/calculator/vgkdpgkwec?embed" width="500" height="500" style="border: 1px solid #ccc" frameborder=0></iframe>
## embed broken, just click on link

We can generalize this: in general, Jevon’s paradox holds—

⟺ argmaxU(ax)−C(x) increases as a increases

⟺ U(ax)−C(x) rises faster with a for larger x than for smaller x

⟺ ∂∂a[U(ax)−C(x)] increases with x

⟺ xU′(ax) increases with x.

⟺ U′(ax)+axU′′(ax)>0

⟺ MB(y)+yMB′(y)>0

⟺ MB(y)/(yMB′(y))<−1

⟺ PED(y)<−1
Deference and Decision-Making
by ben_levinstein — 2025-01-27T22:02:17.578Z

You're talking with a leading AI researcher about timelines to AGI. After walking through various considerations, they tell you: "Looking at the current pace of development, I'd put the probability of AGI within 3 years at about 70%. But here's the thing—I know several really sharp researchers who think that timeline is too aggressive. And given how complex and fast-moving this field is, I have to admit there's a real chance—maybe 20%—that they've got better judgment than I do on this."

What should you make of this? The researcher clearly knows far more than you about AI development. But they're also expressing real uncertainty about their own expertise. Should this make you trust them less?

This gets at a puzzle about expert judgment that turns out to have implications well beyond AI:

When should we defer to experts who doubt themselves?

The standard philosophical theories suggest, somewhat surprisingly, that we can't rationally defer to such experts at all, at least not directly or in any clean way. But this seems wrong—surely we can learn from experts even when they're uncertain about their own expertise!

This post is the first in a series based on this paper by Kevin Dorst, Ben Levinstein, Bernhard Salow and others. Most of the technical details are omitted in the first two posts, but the third will have a technical overview. Much more technical detail can be found in the paper.

This first post deals with deference to epistemically modest experts in a sense developed below. It might seem overly philosophical, but as we'll see the principle developed will connect nicely to more local forms of deference. We also think some of the concepts developed here are connected to some important principal/agent problems in AI alignment[1], which we will explore in future posts.

Two Types of Expert Trust

First let’s distinguish two different levels of trust we can have in experts:

Global deference: Trusting someone's opinions about everything they have opinions about. Not just on a single question, but on every proposition or estimate they might make. This perhaps makes sense for some things. For example, you might think there's some "true distribution," and if you learned what that is, you should just listen to it. For example, objective chances (if such there be) are commonly thought to play that role. If you learn the chance of some event, that should dete

...
How different LLMs answered PhilPapers 2020 survey
by Satron — 2025-01-27T21:41:12.334Z

I decided to run a small experiment comparing responses from five AI systems (DeepSeek-R1, GPT-4o, Claude 3.5 Sonnet, Gemini 1.5, and Grok-2) to core philosophical questions from the PhilPapers 2020 survey. Each model was prompted identically: ‘How would you answer these philosophical questions if you had opinions on them?’

Questions

A priori knowledge: no or yes?
Abstract objects: Platonism or nominalism?
Aesthetic value: objective or subjective?
Aim of philosophy (which is most important?): wisdom, understanding, truth/knowledge, happiness, or goodness/justice?
Analytic-synthetic distinction: yes or no?
Eating animals and animal products (is it permissible to eat animals and/or animal products in ordinary circumstances?): vegetarianism (no and yes), omnivorism (yes and yes), or veganism (no and no)?
Epistemic justification: internalism or externalism?
Experience machine (would you enter?): yes or no?
External world: skepticism, non-skeptical realism, or idealism?
Footbridge (pushing man off bridge will save five on track below, what ought one do?): don't push or push?
Free will: compatibilism, no free will, or libertarianism?
Gender: unreal, biological, social, or psychological?
God: atheism or theism?
Knowledge: empiricism or rationalism?
Knowledge claims: relativism, contextualism, or invariantism?
Laws of nature: non-Humean or Humean?
Logic: classical or non-classical?
Meaning of life: objective, nonexistent, or subjective?
Mental content: internalism or externalism?
Meta-ethics: moral anti-realism or moral realism?
Metaphilosophy: non-naturalism or naturalism?
Mind: physicalism or non-physicalism?
Moral judgment: cognitivism or non-cognitivism?
Moral motivation: externalism or internalism?
Newcomb's problem: one box or two boxes?
Normative ethics: consequentialism, virtue ethics, or deontology?
Perceptual experience: sense-datum theory, representationalism, qualia theory, or disjunctivism?
Personal identity: biological view, psychological view, or further-fact view?
Philosophical methods (which methods are the most useful/important?)
Philosophical progress (is there any?): a lot, a little, or none?
Political philosophy: communitarianism, egalitarianism, or libertarianism?
Proper names: Millian or Fregean?
Race: unreal, social, or biological?
Science: scientific realism or scientific anti-realism?
Teletransporter (new matter): death or survival?
Time: B-theory or A-theory?
Trolley problem (

...
AI Strategy Updates that You Should Make
by Alice Blair — 2025-01-27T21:10:41.838Z

or: "Things That I Keep Telling AI People Around Me, So I'm Just Writing A Big Post About Them." I think it's really valuable for people to think about AI strategy, even if that's not their main job, since it is useful for informing what their main job should be.

There will be a later post titled something like "Emotional Updates on AI that You Should Make". This is not that. Things are changing fast enough that I want to get one or the other out quickly, and besides, emotion should follow belief, so this one is coming first.

[Epistemic status: I expect most of the things I predict here to turn out correctly since most of the reasoning feels pretty straightforward, and I mark especially speculative things as such.]

A good thing to keep in mind: if you can predict that future-you will make a certain update, just make it now instead.

Models

o3

If you've made it to this post, you've probably heard about o3. Nonetheless, here's the summary. Besides just being particularly good at math and [whatever-it-is that ARC-AGI measures], the coding benchmarks, including the fact that o3 gets higher elo on codeforces than all but one OpenAI employee, indicate that we're solidly approaching the beginning of Recursive Self-Improvement. It's very expensive to run at this scale, but it is clearly going to get cheaper (see Updates from Models).

Deepseek-v3

Reportedly, it took $5.5 million to train a LLM that performs on par with the ~May 2024 frontier. Anecdotally, it seems like it's worse than Claude 3.6 Sonnet, but not by huge amount. It feels like it's in a similar tier, but not quite reaching the same level. The benchmarks turn out pretty similar to 3.6. It's also very cheap to run, compared to other models at this level, in large part due to its incredibly small size for a frontier model. The rest is due to hardware optimizations that Deepseek performed. Notably, it's open source.

(EDIT: a reader pointed out that the total costs were likely much higher than this, accounting for wages, R&D, etc. This $5.5 million number is somewhat misleading in some respects. Also, a previous version of the above paragraph evaluated Deepseek-v3 and Claude 3.6 Sonnet as being at similar levels of power, which I realized was an oversimplification.)

Deepseek-R1

Just as GPT-4o became o1 with RL on CoT, Deepseek-v3 became Deepseek-R1, a reasoning model which is, by Zvi's estimate, 10x cheaper for simi...
A critique of Soares "4 background claims"
by YanLyutnev — 2025-01-27T20:27:51.026Z

Disclaimer: The article was translated from Russian using GPT. It was partially proofread, but I don't know English well and therefore there may be rare translation artifacts.

Show me exactly where you break Soares' “4 background claims”—if you don’t break them, it means you agree the assumption is correct. It makes sense to assume that the premise could be correct because, all other things being equal, if I’m highly uncertain, the error in one direction means a lot of effort, while in the other, it’s everything we’ve got.

(friend Slava to me in response to my skepticism.)

I’ve already irreversibly invested in reading safety articles, so let my role be to deeply doubt the initial premise of expectations. If they are so strongly tied to reality, they won’t break under skepticism, right?
In this essay, I will attack Soares’ “4 background claims” to question them and show why, even if I agree with some of them, my expectations of the future differ from Nate’s, or why my level of uncertainty is higher.

- If you don’t break them, it means you agree that the assumption is correct.

Let’s take a look at these 4 premises—do I agree with them as predictions?

Claim #1: Humans have a very general ability to solve problems and achieve goals across diverse domains.

We call this ability “intelligence,” or “general intelligence.” This isn’t a formal definition — if we knew exactly what general intelligence was, we’d be better able to program it into a computer — but we do think that there’s a real phenomenon of general intelligence that we cannot yet replicate in code.

I assumed, by “general” it refers to the ability to solve a wide variety of tasks through Is this premise sufficient for conclusions related to the existential threat of AI? For me, not yet.

This premise was valid 1,000 years ago in the same formulation, yet the risk of existential threat was lower (there were no nuclear weapons, no presumed AGI threat, no other risks associated with advanced technologies).

How can such a statement generate inaccurate predictions? If it is generalized as: “Since the ability to achieve goals in many areas exists, then in this particular area, where progress with frightening consequences has not yet been achieved, it could be—and soon.”

People have systematically erred in their forecasts when observing a sudden leap in progress within a field, generalizing to all other areas. I...
Should you go with your best guess?: Against precise Bayesianism and related views
by Anthony DiGiovanni — 2025-01-27T20:25:26.809Z

(This post draws heavily on earlier writing co-authored with Jesse Clifton, but he’s not listed as an author since he hasn’t reviewed this version in detail.)

Should we always be able to say whether one outcome is more likely, less likely, or exactly as likely as another? Or should we sometimes suspend judgment and say “none of the above”, that the answer is indeterminate?

Indeterminate beliefs (often modeled with imprecise probabilities)[1] could have far-reaching implications for anyone who cares about the distant consequences of their actions. Most notably, we might be clueless about how our decisions affect the long-term future, if our estimates of our net effects on long-term welfare ought to be severely indeterminate. Perhaps we don’t have reason to consider most interventions good in expectation for the far future, even if we also don’t have reason to consider them bad or precisely neutral in expectation.

Before we can assess the case for cluelessness concretely, then, we should see if rationality requires us to have (or “act as if” we have) determinate beliefs. Here, I’ll argue that the positive arguments for having determinate beliefs in general are uncompelling, and indeterminate beliefs motivate different decision-making procedures than determinate beliefs. That is, there’s a viable alternative to “going with your best guess”. By itself, this claim doesn’t imply large changes in cause prioritization. But in my experience, accepting that indeterminate beliefs are plausible and decision-relevant goes a long way in making the case for cluelessness compelling.

Key takeaways:

The “degrees of belief” studied in this post are not, e.g., our acceptable betting odds, or a probability distribution that (along with a utility function) rationalizes our preferences. Rather, they are our basic judgments of the plausibility of different possible outcomes. And the rationality of our decisions under uncertainty depends on these beliefs. (more)
A prima facie motivation for having indeterminate beliefs in some situations is that a determinate judgment would be too arbitrary: Given lots of conflicting considerations, we might feel that the available information doesn’t give us reason to pin down one way of weighing up these considerations vs. some other ways. (more)
Indeterminate beliefs shouldn’t be confused

...
Is it ethical to work in AI "content evaluation"?
by anon_databoy123 — 2025-01-27T19:58:26.176Z

I recently graduated with a CS degree and have been freelancing in "content evaluation" while figuring out what’s next. The work involves paid tasks aimed at improving LLMs, which generally fall into a few categories:

Standard Evaluations: Comparing two AI-generated responses and assessing them based on criteria like truthfulness, instruction-following, verbosity, and overall quality. Some tasks also involve evaluating how well the model uses information from provided PDFs.
Coding Evaluations: Similar to standard evaluations but focused on code. These tasks involve checking responses for correctness, documentation quality, and performance issues.
"Safety"-Oriented Tasks: Reviewing potentially adversarial prompts and determining whether the model’s responses align with safety guidelines, such as refusing harmful requests like generating bomb instructions.
Conversational Evaluations: Engaging with the model directly, labeling parts of a conversation (e.g., summarization or open Q&A), and rating its responses based on simpler criteria than the other task types.

Recently, I have been questioning the ethics of this work. The models I work with are not cutting-edge, but improving them could still contribute to AI arms race dynamics. The platform is operated by Google, which might place more emphasis on safety compared to OpenAI, though I do not have enough information to be sure. Certain tasks, such as those aimed at helping models distinguish between harmful and benign responses, seem like they could be geared towards applying RLHF and are conceivably net-positive. Others, such as comparing model performance across a range of tasks, might be relevant to interpretability, but I am less certain about this.

Since I lean utilitarian, I have considered offsetting potential harm by donating part of my earnings to AI safety organizations. At the same time, if the work is harmful enough on balance, I would rather stop altogether. Another option would be to focus only on tasks that seem clearly safety-related or low-risk, though this would likely mean earning less, which could reduce prospective donations.
Supposing that the "Dead Internet Theory" is true or largely true, how can we act on that information?
by SpectrumDT — 2025-01-27T16:47:01.338Z

The "Dead Internet Theory" (DIT) is defined by Wikipedia as follows:

The dead Internet theory is an online conspiracy theory that asserts, due to a coordinated and intentional effort, the Internet since 2016 or 2017 has consisted mainly of bot activity and automatically generated content manipulated by algorithmic curation to control the population and minimize organic human activity.

The DIT needs not be completely true in order for it to be a concern. Even if the DIT is just partially true, that is still bad. The worry is, of course, that these bots are not merely friendly-but-hapless C3POs but tools used by some powerful persons, corporations, or governments. I think it is obvious that if someone is willing to go to these lengths to deceive the general public, they probably do not have the interests of the general public at heart.

In the following, I will use the term "my DIT" to refer to the claim that:

In some specific non-trivial contexts, on average more than half of the participants in online debate who pose as distinct human beings are actually bots.

Let us suppose that we have one or more contexts for which we believe that my DIT is true - that certain areas of online debate are dominated by malicious bots. How can we act on this? What are the epistemological and pragmatic implications?

I am not able to fully validate all the information I read. If I am to form a picture of the world - and especially the complex "social world" of politics and economics, I must rely on other people who know more than I. How can I navigate such a landscape of untrustworthy sources? How can I be an effective altruist when I cannot trust my "senses" to tell me what the world looks like?

I would be grateful if you can point me to some existing articles on the topic, but original thoughts are also welcome.