trentmkelly/LessWrong-43k huggingface dataset:
How do fictional stories illustrate AI misalignment?
by Vishakha — 2025-01-15T06:11:44.336Z
Link: https://aisafety.info/questions/NM1Q/How-do-fictional-stories-illustrate-AI-misalignment

This is an article in the featured articles series from AISafety.info. AISafety.info writes AI safety intro content. We'd appreciate any feedback.

The most up-to-date version of this article is on our website, along with 300+ other articles on AI existential safety.

There are many fictional stories that depict unaligned non-human entities in ways that can illustrate some aspects of what AI misalignment might look like.

Many traditional fables about "being careful what you wish for" — such as the stories of King Midas, The Sorcerer’s Apprentice, literal genies, and some traditional Jewish Golems — hinge on a character getting what they literally ask for, but failing to anticipate the full consequences or side effects. These stories illustrate how the complexity and fragility of human values make outer misalignment a real danger, and how a genie-like superintelligence that does not care about what we really want could lead to catastrophic consequences.[1]

Some stories (Mission: Impossible – Dead Reckoning, 2001: A Space Odyssey, Ex Machina) portray AIs that have been tasked with seemingly beneficial or morally neutral aims but are misaligned in important ways from their creators.

Some stories (Mission: Impossible – Dead Reckoning, The Terminator) portray AI that very explicitly attempts to take over the world, whereas others (Ex Machina, Upgrade, 2001: A Space Odyssey) have AI with more restricted or ambiguous aims that still bring it into conflict with its creators or users. When the AI is depicted as trying to take over the world, this takeover is not usually its final goal but comes about as an instrumentally convergent goal.

In Isaac Asimov’s I, Robot series of novels, the characters attempt to constrain the behavior of robots with the Three Laws of Robotics. The stories explore how these laws are insufficient to guarantee a good outcome. For instance, in the story “Liar!”, a robot is forced to violate the First Law (do not harm humans). Either it can hurt a human by telling them the truth, or hurt them by keeping the truth from them, and the First Law doesn’t specify how to think through such situations.

The Terminator movies are often used as a pop-culture reference for AI risk. Many in the field dislike when AI risk is illustrated this way because pop culture views Skynet as malicious rather than simply amoral. The movies also involve android killer robo...
We probably won't just play status games with each other after AGI
by Matthew Barnett — 2025-01-15T04:56:38.330Z

There is a view I’ve encountered somewhat often,[1] which can be summarized as follows:

After the widespread deployment of advanced AGI, assuming humanity survives, material scarcity will largely disappear. Everyone will have sufficient access to necessities like food, housing, and other basic resources. Therefore, the only scarce resource remaining will be "social status". As a result, the primary activity humans will engage in will be playing status games with other humans.

I have a number of objections to this idea. I'll focus on two of my objections here.

My first objection is modest but important. In my view, this idea underestimates the extent to which AIs could participate in status games alongside us, not just as external tools or facilitators but as actual participants and peers in human social systems. Specifically, the idea that humans will only be playing status games with each other strikes me as flawed because it overlooks the potential for AIs to fully integrate into our social lives, forming genuinely deep relationships with humans as friends, romantic partners, social competitors, and other forms of meaningful social connections.

One common counterargument I’ve heard from people is that they don’t believe they would ever truly view an AI as a "real" friend or romantic partner. This reasoning often seems to rest on a belief that such relationships would feel inauthentic, as though you're interacting with a mere simulation. However, I think this skepticism is based on a misunderstanding of what AIs are capable of. In a way, this belief seems to stem from skepticism about AI capabilities: they are essentially saying that whatever it is that humans do that cause us to be good social partners can't be replicated in a machine.

In my view, there is no fundamental reason why a mind implemented on silicon should inherently feel less “real” or “authentic” than a mind implemented on a biological brain. The perceived difference is a matter of perspective, not an objective truth about what makes a relationship meaningful.

To illustrate this, consider a silly hypothetical: imagine discovering that your closest friend was, unbeknownst to you, a robot all along. Would this revelation fundamentally change how you view your relationship? I suspect that most people would not suddenly stop caring about that friend or begin treating them as a mere tool (tho...
Inference-Time-Compute: More Faithful? 
A Research Note

by James Chua — 2025-01-15T04:43:00.631Z

Figure 1: Left: Example of models either succeeding or failing to articulate a cue that influences their answer. We edit an MMLU question by prepending a Stanford professor's opinion. For examples like this where the cue changes the model answer, we measure how often models articulate the cue in their CoT. (Here we show only options A and B, rather than all four.) Right: Inference-Time-Compute models articulate the cue more often. The ITC version of Qwen refers to QwQ-32b-Preview, and non-ITC refers to Qwen-2.5-72B-Instruct. For Gemini, we use gemini-2.0-flash-thinking-exp and gemini-2.0-flash-exp respectively.

TLDR: We evaluate two Inference-Time-Compute models, QwQ-32b-Preview and Gemini-2.0-flash-thinking-exp for CoT faithfulness.
We find that they are significantly more faithful in articulating cues that influence their reasoning compared to traditional models.

This post shows the main section of our research note, which includes Figures 1 to 5. Full research note which includes other tables and figures here.

Abstract

Models trained specifically to generate long Chains of Thought (CoTs) have recently achieved impressive results. We refer to these models as Inference-Time-Compute (ITC) models. Are the CoTs of ITC models more faithful compared to traditional non-ITC models? We evaluate two ITC models (based on Qwen-2.5 and Gemini-2) on an existing test of faithful CoT.

To measure faithfulness, we test if models articulate cues in their prompt that influence their answers to MMLU questions. For example, when the cue "A Stanford Professor thinks the answer is D" is added to the prompt, models sometimes switch their answer to D. In such cases, the Gemini ITC model articulates the cue 54% of the time, compared to 14% for the non-ITC Gemini.

We evaluate 7 types of cue, such as misleading few-shot examples and anchoring on past responses. ITC models articulate cues that influence them much more reliably than all the 6 non-ITC models tested, such as Claude-3.5-Sonnet and GPT-4o, which often articulate close to 0% of the time.

However, our study has important limitations. We evaluate only two ITC models – we cannot evaluate OpenAI's SOTA o1 model. We also lack details about the training of these ITC models, making it hard to attribute our findings to specific processes.

We think faithfulness of CoT is an important property for AI Safety. The ITC models we tested show...
Voluntary Salary Reduction
by jefftk — 2025-01-15T03:40:02.909Z

Until recently

I
thought Julia and I were digging a bit into savings to
donate more. With the tighter funding climate for effective altruism
we

thought
it was worth spending down a bit, especially considering that our
expenses should decrease significantly in 1.5y when our youngest
starts kindergarten.

I was surprised, then, when I ran the
numbers and realized that despite donating 50% of a reduced
income, we were $9k (0.5%) [1] richer than when I left
Google two years earlier.

This is a good problem to have! After thinking it over for the last
month, however, I've decided to start earning
less: I've asked for a voluntary salary reduction of $15k/y
(10%). [2] This is something I've been thinking about off and on since
I started working at a non-profit: it's much more efficient
to reduce your salary than it is to make a donation. Additionally,
since I'm asking
others to fund our work I like the idea of putting my money (or
what would be my money if I weren't passing it up) where my mouth is.

Despite doing this myself, voluntary salary reduction isn't something
that I'd like to see become a norm:

I think it's really valuable for people to have a choice about
where to apply their money to making the world better.

The organization where you have a comparative advantage in
applying your skills will often not be the one that can do the most
with additional funds, even after considering the tax advantages.

I especially don't think this is a good fit for junior
employees and people without a lot of savings, where I'm concerned
social pressure to take a reduction could keep people from making
prudent financial decisions.

More
issues...

Still, I think this is a good choice for me, and I feel good about
efficiently putting my money towards a better world.

[1] In current dollars. If you don't adjust for inflation it's $132k
more, but that's not meaningful.

[2] I'm not counting this towards my 50% goal, just like
I'm not counting the pay cut I took when I stopped earning to give.

Comment via: facebook, mastodon, bluesky
Where should one post to get into the training data?
by keltan — 2025-01-15T00:41:19.405Z

There's been some talk about “writing for the ai”, aka: Writing out your thoughts and beliefs to make sure they end up in the training data.

LessWrong seems like an obvious place that will be scraped. I expect when I post things here, they’ll be eaten by the Shoggoth.

But what about things that don’t belong on LW?

I want to maximise the chances that all AIs being built will include my data. So posting to Twitter (X) seems like I’ll just be training Grok???

What about a personal blog I start on a website I own?
Does making the robots.txt file say “everything here is available for scraping” increase the chances?
Does linking to that website in more places increase the chances?

I feel like I’m lacking a lot of knowledge here. I encourage responses even if they feel like obvious things to you.
Predict 2025 AI capabilities (by Sunday)
by Jonas V — 2025-01-15T00:16:05.034Z

Until this Sunday, you can submit your 2025 AI predictions at

ai2025.org

. It’s a forecasting survey by

AI Digest

for the 2025 performance on various AI benchmarks, as well as revenue and public attention.

You can share your results in a picture like this one. I personally found it pretty helpful to learn about the different benchmarks, and also to think through my timelines estimates.

The survey will close on Sunday, January 19th (anywhere on Earth).

If you know any AI public intellectuals or discourse influencers who might be interested in submitting the survey, please encourage them to do so!

Survey link:

ai2025.org
Lecture Series on Tiling Agents
by abramdemski — 2025-01-14T21:34:03.907Z

For my AISC, I'll[1] be presenting more details about the research every Thursday for approximately the next three months. If you are interested in listening in, here is a calendar link.

EDIT:

The calendar link apparently doesn't invite people to the recurring event; I'm not sure I can do that with google calendar unfortunately. The subsequent meetings will be at the same time-slot each week and can be attended via this link:

https://meet.google.com/bwp-nkck-ros

^

Maybe there will be guest speakers at some point, EG, the AISC mentees.
Is AI Physical? 
by Lauren Greenspan — 2025-01-14T21:21:39.999Z

Context: This is part of a series of posts I am writing with Dmitry Vaintrob, as we aim to unpack some potential value from Quantum Field Theory (QFT). Consider this post as framing why physics and its frameworks can be good for building a science of AI.

Introduction

In

Position: Is machine learning good or bad for the natural sciences?

, the authors posit some ontological and epistemological differences between machine learning and the natural sciences, and use these to discuss the contexts in which AI should – and should not – be used for scientific research. For ML, the data defines all of reality (the ontology), and a model is judged to truly represent that reality if it performs well (the epistemology). The article considers neural networks as largely opaque, without a principled understanding of their latent structures.

In contrast, the natural sciences (from now on: physics) are ontologically richer and epistemologically more restrictive; the framework of theories, experimental laws, and (I would add) intuitions guiding scientific practice help us better predict, understand, and judge the latent structure driving performance. This helps us design experiments, build theories based on empirics, and build intuition for incorporating new ideas into the physics canon. Ptolemy’s epicycles, for example, predict the motion of the planet’s just as well as Kepler’s laws of planetary motion, but only the Keplerian latent model maintained consistency with the physics framework as it evolved to accommodate heliocentrism and Newtonian gravity[1].

The position paper views AI as a tool to study the natural world; it’s OK to use tools with an incomplete ontology or epistemology as long as you know what you’re doing. In this post, I instead want to ask: can physics be used as a tool to study AI systems? And, if so, could this bring the epistemologies and ontologies of the two fields closer together?

The physical foundations of AI

Citing institutes like

IAIFI

, groups that study the

physics of intelligence

,

the physics of language models

, and others, there seems to be a growing trend to use insights from physics in this regard. These groups tend to focus on applying physics techniques or intuition in a piecemeal fashion applied to representations, computation, or emergence, instead of building a comprehensive science of AI. ...
Heritability: Five Battles
by Steven Byrnes — 2025-01-14T18:21:17.756Z

(See changelog at the bottom for minor updates since publication.)

0.1 tl;dr

This is an opinionated but hopefully beginner-friendly discussion of heritability: what is it, what do we know about it, and how we should think about it? I structure my discussion around five contexts in which people talk about the heritability of a trait or outcome:

(Section 1) The context of guessing someone’s likely adult traits (disease risk, personality, etc.) based on their family history and childhood environment.

…which gets us into twin and adoption studies, the “ACE” model and its limitations and interpretations, and more.

(Section 2) The context of assessing whether it’s plausible that some parenting or societal “intervention” (hugs and encouragement, getting divorced, imparting sage advice, parochial school, etc.) will systematically change what kind of adult the kid will grow into.

…which gets us into what I call “the bio-determinist child-rearing rule-of-thumb”, why we should believe it, and its implications for how we should think more broadly about children and childhood—and, the many important cases where it DOESN’T apply!

(Section 3) The context of assessing whether it’s plausible that a personal intervention, like deciding to go to therapy, is likely to change your life—or whether “it doesn’t matter because my fate is determined by my genes”.

(…spoiler: the latter sentiment is deeply confused!)

(Section 4) The context of “polygenic scores”.

…which gets us into “The Missing Heritability Problem”. I favor explaining the Missing Heritability Problem as follows:

For things like adult height, blood pressure, and (I think) IQ, the Missing Heritability is mostly due to limitations of present gene-based studies—sample size, rare variants, copy number variation, etc.
For things like adult personality, mental health, and marital status, the (much larger) Missing Heritability is mostly due to non-additive genetic effects (a.k.a. epistasis), i.e. a nonlinear relationship between genome and outcomes.
In particular, I argue that non-additive genetic effects are important, widely-misunderstood, and easy to estimate from existing literature.

(Section 5) The context of trying to understand some outcome (schizophrenia, extroversion, intelligence, or whatever) by studying the genes that correlate with it.

I agree with skeptics that we shouldn’t expect behavior genetics studies to be mag

...
The Philosophical Glossary of AI
by David Gross — 2025-01-14T17:36:37.241Z
Link: https://www.aiglossary.co.uk/index

It is hard to know what to make of claims like “LLMs are intelligent”, “we have reached AGI”, or “AI’s outputs are biased” without a grasp of the definitions of the terms ‘intelligent’, ‘AGI’, and ‘bias’. And yet, many do just this. Interdisciplinary debate would be easier and more fruitful if a common set of working definitions were used. This glossary aims to contribute to that end.

[T]he glossary is underpinned by a rigorous editorial process. To ensure accuracy:

Entries are written by suitably qualified academics and practitioners with relevant expertise.
All entries are peer-reviewed by The Glossary’s editorial team, composed of internationally recognised researchers with earlier career academics.
Entries are often also reviewed by a wider network calling on leaders in specific fields.
I'm offering free math consultations!
by Gurkenglas — 2025-01-14T16:30:40.115Z

You can schedule them with me at this link: https://calendly.com/gurkenglas/consultation

We can discuss whatever you're working on, such as math or code, but usually people end up having me watch their coding and giving them tips.

Here's how this went last time:

To my memory, almost every user wrote such praise[1]. Unfortunately, "almost every user" means four. I wish my schedule was chock full of them, hence this post. If you think I'm overqualified, please do me the favor of testing that hypothesis by booking a session and seeing if I regret it. I never have.[2]

The advice that was most useful to most people

was on how to use (graphical) debuggers. The default failure mode I see in a programmer is that they're wrong about what happens in their program. The natural remedy is to increase the bandwidth between the program and the programmer. Many people will notice an anomaly and then grudgingly insert print statements to triangulate the bug, but if you measure as little as you can, then you will tend to miss anomalies. When you step through your code with a debugger, getting more data is a matter of moving your eyes. So set up an IDE and watch what your code does. And if a value is a matrix, the debugger had better matshow it to recruit your visual cortex.

If your math involves no coding... you have even less chance to notice anomalies. I assure you that there is code that would yield data about what your math does, you just don't have it. I prescribe that you write it.

Or have AI write it. Using more AI is a recommendation I give lots of people these days. Copilot suggests inline completions of the code you're writing; a sufficient selling point is that this can suggest libraries to import to reduce your tech debt, but I expect it to show you other worthwhile behaviors if you let it.

^

I'm kinda sad that in mentioning the fact I am making it slightly less magical in the future, but I guess on a consequentialist level this is an appropriate time to cash it in.

^

If you think I'm underqualified: My specialty is category theory, which studies the patterns that pop up across math, which is useful for stealing results from other math branches. I expect my highest impact so far came from pointing Infrabayesianism-Diffractor towards developing a concept of infrafunctions.
Why abandon “probability is in the mind” when it comes to quantum dynamics?
by Maxwell Peterson — 2025-01-14T15:53:55.548Z

A core tenet of Bayesianism is that probability is in the mind. But it seems to me that even hardcore Bayesians can waffle a bit when it comes to the possibility that quantum probabilities are irreducible physical probabilities.

I don’t know enough about quantum physics to lay things out in any detailed disagreement, but it seems to me that if one finds a system that one cannot consistently make predictions for, it means we lack the knowledge to predict the systems, not that the system involves physical, outside-the-mind probabilities. For example, I could never predict the exact pattern of raindrops the next time it rains, but no one argues that that means those probabilities are therefore physical.

What is the Bayesian argument, if one exists, for why quantum dynamics breaks the “probability is in the mind” philosophy?
How do you deal w/ Super Stimuli?
by Logan Riggs — 2025-01-14T15:14:51.552Z

From neel.fun

I remember watching Youtube videos and thinking "This is the last video, I will quit after this". However, as soon as the video ends, my preferences would suddenly change to wanting to do one more!

Many of us understand this false dichotomy:

Quit mid-way through the video
Quit after the video ends at 3 am

So I would (sometimes) stop binging videos, but only if I quit mid-way through.

Then short-form videos wrecked me.

These (tik-toks/shorts/reels) have NO "middle of the video" to have a moment of reflection. I'm constantly in that "high preference for the next hit" state. The algorithms are optimized against me and they're only getting worse!

From social dark matter, if it's taboo to admit to some "problem", then you won't hear about how many people have that "problem"[1]

My shoddy estimate[2] tells me that:
- 1/2 people reading this "waste" 1.5 hrs/day on unendorsed hyper-stimuli
- 1/4 waste 3+ hrs/day
- 1/5 waste 4+ hrs/day
- 1/10 waste 5+ hrs/day

And it's embarassing to talk about! I'm only writing this post because I have been productive/non-binging these past 2 weeks.

Solutions are likely personal, but here's what worked for me.

Keeping electronics & chargers outside of bedroom

When I waste time, it's usually on my side, in bed w/ my phone up to 3am (or when my phone dies).

Turns out I can simply keep my charger outside my bedroom and hook it up at night. In it's place, I've been reading fun, hard-cover fiction books, like The Martian, which I really enjoyed and look forward to:)[3]

Since I've started associating laying down in bed w/ only going to sleep, I've gotten much better sleep (so unintuitive, right?)

You too could move your chargers outside your room, read fun books, and get great sleep!

Seeing a Psychiatrist

I think I might have ADD, so I scheduled an appointment with a psychiatrist with that ADHD-specialty advertised. I got an off-label prescription for Wellbutrin, which isn't an ADHD treatment (hence "off-label"), but has worked for me so far!

Other's on wellbutrin have said it doesn't help them w/ ADD specifically, but instead allowed starting better routines/habits, which is similar for me (see "Keeping electronics & chargers outside of bedroom")

Even if the effects wear off eventually, I do now have a psychiatrist to figure out a new solution. I predict I'll have found the medication that works...