trentmkelly/LessWrong-43k huggingface dataset:
What college major should I choose if I am unsure?
by contrejour — 2025-05-28T17:50:57.899Z

First and foremost, it is important to understand that I finished my Associate's degree. I was about to finish my linguistics major because my original plan was to "move abroad and teach English." That was when I was 17. During that time, I decided to change my major and asked my advisor, who told me explicitly that that was not possible due to how close I was. I made the decision to transfer to another university to choose a major that was broader than linguistics. That is when I reach Economics as a major. It is a social science, and also regarded as STEM-like. However, I am not a fan of mathematics as a subject. I'm quite terrible at it. That is when other options appeared, such as PoliSci or History.

Nevertheless, I'm still quite unsure.

Part of me resonates more within the humanities, art, philosophy, history, everything that has to do with the creative sciences. However, in the long run, those majors do not always create ROI (return of investment), and despite my interests, I want to major in something that also helps me live day to day.

What major should I choose then? What pathway should I take? What should I do?
Can genetic privilege exist within an evolutionary equilibrium?
by AynonymousPrsn123 — 2025-05-28T17:44:21.263Z

ChatGPT and evolutionary biologists all seem to agree that the answer to my question is a resounding "yes," but I don't understand why. In particular, if a subpopulation has a persistent genetic privilege, doesn't that inherently imply that the population is NOT at an evolutionary equilibrium?

Please explain this to me using a mathematical model with reasonable and true assumptions. Alternatively, you could also just provide a "hand-wavey" math-sounding explanation; but please make sure it's logically valid and the assumptions are accurate.
Evaluation As Feedback Cycle
by belos — 2025-05-28T17:02:37.233Z
Link: https://bestofagreatlot.substack.com/p/evaluation-as-feedback-cycle

This post on Best Of A Great Lot is a part of a series on the subject of designing a new form of governance. Each piece aims to stand alone, but fits together on the

Table of Contents

.

Previous:

A Sketch of Belocracy

. Next: Generating Ideas.

We know the path by which optimizers find effectiveness: they follow feedback loops on a signal of quality. This is a core truth of capitalism, evolution, machine learning, our most successful humans and other systems that optimize in response to feedback. Belocracy aims to join these ranks by having an evaluation step and then providing direct incentives for succeeding at that evaluation step.

As people and as a society, we are constantly evaluating: journalism, academic research, political arguments and individual conversations at least sometimes refer to evidence we have about how well our decisions have panned out. But these informal, noncentralized evaluations serve as only a weak feedback mechanism compared to what formal, centralized, independent evaluations can be. Within belocracy, policies are reviewed to determine whether they are meeting their goals, and programs -- whether regulatory or infrastructure -- are evaluated on how well they are succeeding in comparison to their competitors. Rewards and reputation are handed out for successful outcomes. Individuals pursuing their own benefit drives improvement in the system, and as long as the feedback offered is closely aligned with reality, a belocratic system will improve on the back of these rewards.

It's very tempting to delay talking about evaluation until later. Intuitively, the evaluation of a thing comes after the thing itself, so it might seem obvious to talk about how policies and programs are made before talking about how they're evaluated. But the feedback loop of belocracy is the heart of what gives it a chance of letting us make better policies and run our infrastructure more effectively. Without a feedback loop, the decision of what to do always comes down to what sounds better politically or other, worse feedback loops like prestige and money, rather than what works better in practice. So I'm going to describe the evaluation system first and defer to later some parts of the system that evaluation depends upon.

A system of evaluation carries inherent risk: it can become a target for ideologues and opportunists who want power. Supreme Court seats and the ...
How much might AI legislation cost in the U.S.?
by will rinehart — 2025-05-28T16:21:33.860Z

This piece was previously published on my Substack.

Policymakers are rushing to regulate artificial intelligence (AI), but the economic impact of these regulations remains largely unexplored. While

the European Union

and

the United Kingdom

have produced cost estimates, recent developments in the United States offer important new benchmarks. Recent amendments to the California Consumer Privacy Act (CCPA) and regulations implementing President Biden’s Executive Order on AI offer crucial insights into what businesses might expect to pay for compliance. The financial burden could be substantial, running into billions of dollars across the economy. Especially as states push to adopt AI bills, understanding these costs is essential for crafting regulations that balance innovation, safety, and economic viability.

Still, these compliance cost estimates

are notoriously unreliable

. As an alternative approach, I tested whether large language models (LLMs) could provide more realistic estimates by simulating compliance scenarios in the final section of this post. I prompted ChatGPT, Claude, and Grok to act as compliance officers at companies subject to new CCPA provisions and a Bureau of Industry and Security (BIS) rule, asking each to estimate hours needed for first-year implementation and ongoing compliance.

The big takeaways:

For California's risk assessment regulation, Claude and Grok project 400-580 hours will be needed for the first-year of compliance (vs. the official 120 hours) and 150-240 hours thereafter (vs. the official 18-36 hours annually). ChatGPT estimates the time at 90-250 hours initially and 40-150 hours for each additional year.
For the automated decision-making provision of the CCPA, Claude and Grok project 450-730 hours for first-year compliance, far exceeding the official 360-hour estimate. While ChatGPT suggests lower initial costs (80-300 hours), all three LLMs predict significantly higher ongoing annual costs than official projections.
For the BIS reporting rule, Claude projects 1,140 hours for first-year compliance (3x the official estimate of 333 hours), while ChatGPT estimates 280 hours and Grok projects 380 hours. All three LLMs agree ongoing compliance will require substantial resources.

In short, the LLMs consistently predicted much higher compliance costs than the official sources, suggesting a systematic underestimation.

The following analysis ...
What LLMs lack
by p.b. — 2025-05-28T16:19:37.663Z

Introduction

I have long been very interested in the limitations of LLMs because understanding them seems to be the most important step to getting timelines right.

Right now there seems to be great uncertainty about timelines, with very short timelines becoming plausible, but also staying hotly contested.

This led me to revisit LLM limitations and I think I noticed a pattern that somehow escaped me before.

Limitations

To recap, these seem to be the most salient limitations or relative cognitive weaknesses of current models:

System 2 thinking: Planning, see the ongoing weird difficulty to get it to play TicTacToe perfectly or block world, chess, anything that has not been subject of a lot of reasoning RL.

Dealing with new situations: Going out of distribution is a killer for all things DL.

Knowledge integration: Models don't have automatic "access" to skills learned from separate modalities. Even within the same modality skills are not robustly recallable, hence the need for prompting. Also related: Dwarkesh's question.

Learning while problem solving: Weights are frozen and there is no way to slowly build up a representation of a complex problem if the representations that have already been learned are not very close already. This is basically knowledge integration during inference.

Memory: RAGs are a hack. There is no obvious way to feed complex representations back into the model, mostly because these aren't built in the first place - the state of a transformer is spread over all the token and attention values, so recomputing those based on the underlying text is the go-to solution.

Objectivity: See hallucinations. But also self-other/fact-fantasy distinction more generally.

Agency: Unexpectedly we got very smart models that are not very good at getting stuff done.

Cognitive control: The inability to completely ignore irrelevant information or conversely set certain tenets absolute leads to jailbreaks, persistent trick question failures and is also a big part of the unreliability of models.

One category

These seem like a mixed bag of quite different things, but I recently realised that they all belong to the same class of cognitive abilities: These are all abilities that in humans are enabled by and in fact require consciousness.

Is "cognitive abilities enabled by consciousness" maybe a bit tautological? Unconscio...
Playlist Inspired by Manifest 2024
by Commander Zander — 2025-05-28T16:03:29.051Z
Link: https://open.spotify.com/playlist/4rdvKoNg8WjGrS4SVip0RT

Okay, I think it's time to stop polishing this playlist & post it. This is music about the moods of people I met at Manifest. I'm pretty delighted with it!
AISN #56: Google Releases Veo 3
by Corin Katzke — 2025-05-28T16:00:48.187Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-56-google-releases

Welcome to the AI Safety Newsletter by the

Center for AI Safety

. We discuss developments in AI and AI safety. No technical background required.

In this edition: Google released a frontier video generation model at its annual developer conference; Anthropic’s Claude Opus 4 demonstrates the danger of relying on voluntary governance.

Listen to the AI Safety Newsletter for free on

Spotify

or

Apple Podcasts

.

Subscribe to receive future versions.

Google Releases Veo 3

Last week, Google made several

AI announcements

at I/O 2025, its annual developer conference. An announcement of particular note is

Veo 3

, Google’s newest video generation model.

Frontier video and audio generation. Veo 3 outperforms other models on

human preference benchmarks

, and generates both audio and video.

Google showcasing a video generated with Veo 3. (

Source

)

If you just look at benchmarks, Veo 3 is a substantial improvement over other systems. But relative benchmark improvement only tells part of the story—the absolute capabilities of systems ultimately determine their usefulness. Veo 3 looks like a marked qualitative improvement over other models—it generates video and audio with extreme faithfulness, and we recommend you

see

some

examples

for yourself. Veo 3 may represent the point video generation crosses the line between being an interesting toy and being genuinely useful.

Other announcements at I/O 2025. Other highlights from the conference include:

Gemini 2.5

Pro now leads LMArena and WebDev Arena. Deep Think mode, a reasoning feature that scored 49.4% on the USA Mathematical Olympiad 2025 (more than twice OpenAI’s o3, which scored 21.7%). Gemini 2.5 Flash now performs better across reasoning, multimodality, code, and long context while becoming 20-30% more efficient in token usage.

Gemini Diffusion

, an experimental (non-frontier) text diffusion model, delivers output 4-5 times faster than comparable models while rivaling the performance of models twice its size. Most LLMs are autoregressive models, which generate one token at a time—in contrast, diffusion models generate an entire response at once.
Google also announced

Gemma 3n

, an open model small enough to run on mobile devices, a public beta for Google’s autonomous coding agent

Jules

, a new

AI search

feature, an

AI watermarker

that identifies content generated by Google’s systems, and more.

AI is here to stay. AI use is sometimes driven by t...
How Self-Aware Are LLMs?
by Christopher Ackerman — 2025-05-28T12:57:37.998Z

An interim research report

Summary

We introduce a novel methodology for quantitatively evaluating metacognitive abilities in LLMs
We present evidence that some frontier LLMs introduced since early 2024 - but not older or smaller ones - show some metacognitive abilities
The metacognitive abilities that current LLMs do show are relatively weak, and manifest in a context-dependent manner; the models often prefer to use heuristics
Analysis of the probabilities LLMs assign to output tokens provides evidence of an internal signal that may be used for metacognition
There appears to be a dissociation in LLM performance between recognition and recall, with the former supplying a much more robust signal for introspection

Introduction

A basic component of self-awareness, and one amenable to testing, is knowledge of one’s own knowledge. Early work in previous generations of LLMs found that the models’ implicit confidence in the correctness of their outputs, as indicated by the probabilities their output layer assigned to generated tokens, was correlated with the probability of their outputs being correct (i.e., was “calibrated”), suggesting the existence of internal signals that could be used as the basis for self-knowledge.

But can models actually access that information? Do they explicitly “know” what they know? As models have grown larger, researchers have found that models can be fine-tuned to report explicit confidence ratings in their answers that are well calibrated, and that more recent models trained with reinforcement learning from human feedback (RLHF), like ChatGPT, can give calibrated verbal reports of certainty even without specific fine tuning.

A priori this seems somewhat surprising. The transformer architecture underlying modern LLMs is entirely feedforward, offering no opportunity for the reflective processes that underlie the subjective experience of human introspection, and the standard next-token-prediction pretraining task affords no basis for self-modeling. It’s conceivable that targeted RLHF post-training could induce a sort of self-simulation in models (certainly model developers have an incentive to encourage models to differentiate factual knowledge from guesswork to reduce hallucinations). In fact, researchers have recently claimed success in fine-tuning just such a self-simulation mechanism into GPT4 and GPT4o.

Still, caution is war...
Can We Hack Hedonic Treadmills?

by Vincent Li — 2025-05-28T11:42:55.193Z

During a visit to a Hong Kong children’s welfare home, I met a 12-year-old girl I'll call Kylie. She had suffered a severe illness that left her blind, deaf, non-verbal, and nearly immobile, yet no identified damage was done to her brain.

The staff described her, without hesitation, as “always cheerful,” and indeed she smiled the entire time I watched her on the standing frame.

We can't objectively verify Kylie’s internal state, but her case crystallized a puzzle: if the absence of sight, hearing, or mobility doesn't preclude happiness, and neither wealth nor fame can guarantee it, then what truly determines our level of happiness?

The treadmill, uphill and down

Brickman & Campbell (1971) called this puzzle the Hedonic Treadmill: as a person experiences a positive emotional event, expectations and desires rise in tandem which cancels their net long-term impact, resulting in no permanent gain in happiness.

So no matter how hard one tries to gain in happiness, one will remain in the same place.

Later studies show the same homeostasis after negative shocks: people recover from bereavement, disability, or loss, drifting back towards a stable baseline level of happiness. The baseline level may be identified at different points among individuals (such as neutral or positive), but is stubbornly stable within them.

The implication is disheartening. If no action durably lifts well-being, then our daily efforts seem to generate motion without progress.

Can we raise the platform?

We can break the treadmill, downward: substance abuse, chronic depression, or traumatic brain injury can lower one's baseline for years. The biological mechanisms involve long-term changes in dopaminergic and serotonergic signalling, glucocorticoid cascades, and possibly neuroinflammation—none pleasant.

But the platform can surely be raised as well, right? Some empirical literature says yes: subjective well-being is correlated to durable improvements in happiness baseline, suggesting interventions such as:

pursue meaningful, attainable goals
reframe experiences positively
maintain sleep, exercise, diet
invest in close relationships
cultivate realistic self-esteem

The list goes on, and initially it made me hopeful. It painted a path towards a richer, more fulfilling life. But eventually, it felt like something was missing. I haven't found any particular novel hacks, and any sensible person would have known...
Knowledge and Provability Inclusion
by milanrosko — 2025-05-28T10:50:21.686Z

Introduction: The Vanishing Sky Analogy

Consider the following analogy:

In a universe undergoing accelerating expansion, distant galaxies gradually slip beyond our horizon, until the point at which their light can no longer reach us. They become causally disconnected from us: their light will never reach us, and thus they are forever beyond our observation[1].

In a far-distant future, this could lead to a hypothetical situation in which only a single galaxy—our own—remains within the observable universe. All other structures would have vanished beyond the horizon, obscured by the expansion of space. For future observers, the universe would appear to consist solely of this one galaxy. Their cosmology would necessarily be confined to what lies within this narrow observational horizon.

Such a cosmology would be internally consistent, yet fundamentally incomplete—without this incompleteness being apparent.

This raises the question: Since our cosmological models are constructed entirely from what we can observe, is it possible that we, too, are in a similarly constrained position today? If vast regions of the universe already lie beyond our cosmological horizon, then we lack not only access to their information—but even awareness of their absence. We may suspect that our picture of the universe is incomplete, but we cannot know in what way, or to what extent.

Provability Inclusion implies that, despite epistemic limitations, every entity, system, or cosmology is compelled to treat available data as a kind of Löbian Ground Truth[2]—not because it chooses an antecedent freely, but because it is constrained by an enforced (as in forcing[3]) relative consistency.

A tribe living deep within a dense forest might naturally assume that the entire world is forest. However, using the example of an insect confined to a single tree—believing that tree to be the whole of existence—does not validate or invalidate the tribe’s limited perspective.

The Theorem of Provability Inclusion is a limitative formal result in first-order logic (FOL) that helps us look "downward" and recognize ourselves in the insect.

Generalized Common Knowledge

One will quickly notice that provability inclusion presents us with statements that often appear philosophical or paradoxical. This is primarily due to its reliance on the nested interplay of proof predicates, self-reference, and diagonalization. A close co...
AI’s goals may not match ours
by Algon — 2025-05-28T09:30:36.930Z

Context: This is a linkpost for https://aisafety.info/questions/NM3I/6:-AI%E2%80%99s-goals-may-not-match-ours

This is an article in the new intro to AI safety series from AISafety.info. We'd appreciate any feedback. The most up-to-date version of this article is on our website.

Making AI goals match our intentions is called the alignment problem.

There’s some ambiguity in the term “alignment”. For example, when people talk about “AI alignment” in the context of present-day AI systems, they generally mean controlling observable behaviors like: Can we make it impossible for the AI to say ethnic slurs? Or to advise you how to secretly dispose of a corpse? Although such restrictions are sometimes circumvented with "jailbreaks", on the whole, companies mostly do manage to avoid AI outputs that could harm people and threaten their brand reputation.

But "alignment" in smarter-than-human systems is a different question. For such systems to remain safe in extreme cases — if they become so smart that we can’t check their work and maybe can’t even keep them in our control — they'll have to value the right things at a deep level, based on well-grounded concepts that don’t lose their intended meanings even far outside the circumstances they were trained for.

Making that happen is an unsolved problem. Arguments about possible solutions to alignment get very complex and technical. But as we’ll see later in this introduction, many of the people who have researched AI and AI alignment on a deep level think we may fail to find a solution, and that may result in catastrophe.

Some of the main difficulties are:

We can’t see what an AI values, because current AI is not designed in the same way as a web browser, an operating system, or a word processor — rather, it is “grown”. Human programmers design the process that grows the AI. But that process consists of vast numbers of computations that automatically make huge numbers of small adjustments, based on what works best on each piece of training data. The result is something like an alien artifact made of skyscraper-sized spreadsheets full of numbers. We can see what all the numbers are, and we know they have some sort of patterns in them that result in sensible tax advice and cookie recipes. We just understand little about what those patterns are or how they work together. In that sense, we’re looking at a “black box”. And

...