trentmkelly/LessWrong-43k huggingface dataset:
[Cross-post] Every Bay Area "Walled Compound"
by davekasten — 2025-01-23T15:05:08.629Z

[Cross-posted from my substack, davekasten.substack.com.]

(With apologies and thanks to the incomparable

Scott Alexander

,

Richard Ngo

,

Ricki Heicklen

, and Emma Liddell).

Every Bay Area “Walled Compound”

You’re in Berkeley unexpectedly. You’d hoped [1]this would be the one trip through SFO this year where you didn’t end up at

your favorite conference venue

(

previously

,

previous-er-ly

), but, well, life had other plans. Your vacation plans fell through in an unexpected way, and in addition to being an excellent conference venue, and a target of the

odd conspiracy theory by journalists with poor fact-checking skills

, Lighthaven sometimes offers

surprisingly affordable

housing last-minute, for those in the know.

The next morning, you wake up in your stylish bedroom.

You wander through the common area, and see three individuals in monk’s robes congregated around the kitchen island. The seniormost-seeming monk is interrogating the juniormost about their

strategy for fairly cutting the chocolate cake

in front of them and distributing the slices.

You’ll later learn, appropriately, that the juniormost monk was

tricked into coming to Lighthaven

by means of a

single, inaccurate electronic message

to volunteer to assist with planning their

Petrov Day

commemorations

.[2]
After all, it’s better to rein in help, then serve some leaven.

You amble out to the beautiful secret garden. (So secret that you, ah, literally missed it on your first trip to Lighthaven, despite it being approximately 30% of the total grounds. Look, there were some good talks going on, and you were busy.)

Photo credit: Lighthaven. This tree is a great place to hold a Kabbalat Shabbat underneath, incidentally.

Canada geese fly overhead, honking. Your inner Ohioan

notices that you are confused

; it’s the wrong season for them to have migrated this far south, and they’re flying westwards, anyways.

A quick Google discovers that some Canada geese have now established themselves

non-migratorily

in the Bay Area:

The

Migratory Bird Treaty Act

of 1918 banned hunting or the taking of eggs without a permit. These protections, combined with an increase in desirable real estate—parks, golf course and the like—spurred a dramatic turnaround for the species. Canada geese began breeding in the Bay Area—the southern end of their range – in the late 1950s.

You nod, approvingly; this clearly is another part of the East Bay’s well-kn...
Writing experiments and the banana escape valve
by Dmitry Vaintrob — 2025-01-23T13:11:24.215Z

[Note: This is not alignment-related, but rather a spacefiller personal blog post.]

I've been trying to write a public post every day of January. So far I’ve been enjoying it. I don’t think this approach works for everyone: in particular, I’ve also been hanging on by a thread to the schedule and to the ability to sleep. I publicly committed to write 3 posts a week, but my “secret” personal goal to write a post every day for the month. Not only in order to have more output, but because in the past intense daily deadlines have worked pretty well in the past; also there’s a kind of scientific idea of “you’ll get more data on the success of a new behavior if you go the whole hog”.

In order to have a clear “success criterion”, my rules are that

By day n of January (for n≤31), I want to have authored or coauthored n posts or shortforms in January (including the first announcement post).

Here the “end of the day” is defined as Pacific “

Random standard time

” – so January 1st ends at 6AM January 2nd.

During any 7-day period, I’ve written at least 3 posts (this is unlikely to lapse before the previous condition, but is there to avoid cheating by something like “writing 31 short posts in one day”).
There is no restriction on quality, length or content (other than a vague “have an internal sense that you are not hurting the commons by posting overly personal or unproductive schlock on a public forum. Graded leniently, ok?). In my personal precommitment on fatebook, I gave myself a 5% chance of success. It’s a bit higher now but still under 50%.

I’ve actually done a couple of experiments of this type before, and have enjoyed them. The latest was three years ago, when I gave myself a “daily writing minimum” of 12 pages in a small notebook, for a month. It was a conference-heavy month and while at first the pages had discussions of literature and math and productivity:

The last page looked like this:

For projects like this one, I feel like it's very useful to have consistent practices to phone it in: graceful failure modes that allows you to retain steam. In the context of this sprint, I actually rough-drafted a couple of escape valves (including this post) in advance, in case of an emergency backup.

Other practices I've used this sprint:

converting old drafts or conversations into low-effort posts
writing what are essentially math lectures
taking the first idea or coherent set o

...
MONA: Managed Myopia with Approval Feedback
by Seb Farquhar — 2025-01-23T12:24:18.108Z

Blog post by Sebastian Farquhar, David Lindner, Rohin Shah.
It discusses the paper MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hacking by Sebastian Farquhar, Vikrant Varma, David Lindner, David Elson, Caleb Biddulph, Ian Goodfellow, and Rohin Shah.

Our paper tries to make agents that are safer in ways that we may not be able to evaluate through Myopic Optimization with Non-myopic Approval (MONA).

Suppose we know that AI systems will do bad things but we also know that we won’t be able to tell when they do. Can we still prevent unintended behaviour?

In particular cases and ways: yes. Specifically, we show how to get agents whose long-term plans follow strategies that humans can predict without relying on the assumption that humans can notice when the agent tries to use other strategies. In exchange, we give up on having agents that have incomprehensible plans that happen to be really good, but we think this is likely a worthwhile trade for sufficiently superhuman systems.

As a motivating metaphor, imagine a spotlight in the night. So long as an agent remains inside the spotlight, we know how to evaluate what it does. If it strays outside the spotlight, we can only guess whether its actions are good or bad. Scalable oversight tries to make the spotlit area bigger so that we can evaluate more kinds of actions. Instead, we try to make sure the agent is trained in a way that does not incentivize leaving the spotlight.

Reducing multi-step reward hacking to single-step reward hacking

Our approach is based on two key assumptions: (a) it is easier to evaluate single actions than it is to evaluate longer-term trajectories, and (b) single-step reward hacks are typically less dangerous than multi-step reward hacks. Because we believe this, we believe that finding a way to produce an agent that will engage in single-step reward hacking but which will not do multi-step reward hacking is progress.

The basic reason single actions are easier to evaluate is that they come with a smaller decision-space. A single action gives the agent fewer options for undermining our evaluations or tampering with rewards or sensors. Additionally, multi-step plans allow the agent to use early actions to enter states that are very different from any states that humans have ever experienced. This lack of familiarity makes it harder to know what the impact of a...
How useful would alien alignment research be? 
by Donald Hobson — 2025-01-23T10:59:22.330Z

Imagine aliens on a distant world. They have values very different to humans. However, they also have complicated values, and don't exactly know their own values.

Imagine these aliens are doing well at AI alignment. They are just about to boot up a friendly (to them) superintelligence.

Now imagine we get to see all their source code and research notes. How helpful would this be for humans solving alignment?
What are the differences between AGI, transformative AI, and superintelligence?
by Vishakha — 2025-01-23T10:03:31.886Z
Link: https://aisafety.info/questions/5864/What-are-the-differences-between-AGI,-transformative-AI,-and-superintelligence

This is an article in the featured articles series from AISafety.info. AISafety.info writes AI safety intro content. We'd appreciate any feedback.

The most up-to-date version of this article is on our website, along with 300+ other articles on AI existential safety.

These terms are all related attempts to define AI capability milestones — roughly, "the point at which artificial intelligence becomes truly intelligent" — but with different meanings:

AGI stands for "artificial general intelligence" and refers to AI programs that aren't just skilled at a narrow task (like playing board games or driving cars) but that have a kind of intelligence that they can apply to a similarly wide range of domains as humans. Some call systems like Gato AGI because they can solve many tasks with the same model. However, the term is more often used for systems with at least human-level general competence, so more typically AGI is still seen as a potential future development.[1]
Transformative AI is any AI powerful enough to transform society. (The term is unrelated to the transformer architecture.) Holden Karnofsky defines it as AI that causes at least as big an impact as the Agricultural or Industrial Revolutions, which increased economic growth many times over. Ajeya Cotra's "Forecasting TAI with biological anchors" describes a "virtual professional", i.e., a program that can do most remote jobs, as an example of a system that would have such an impact.

Superintelligence is defined by Nick Bostrom as "any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest". This is by far the highest bar out of all the concepts listed here, but it may be reached a short time after the others, e.g., because of an intelligence explosion.

Other terms which are sometimes used include:

Advanced AI is any AI that's much more powerful than current AI. The term is sometimes used as a loose placeholder for the other concepts here.
Human-level AI is any AI that can solve most of the cognitive problems an average human can solve. Current AI has a very different profile of strengths and weaknesses than humans, and this is likely to remain true of future AI: before AI is at least human-level at all tasks, it will probably be vastly superhuman at some important tasks while still being weaker at others.
Strong AI was defined by John Searle as the philosophic

...
Why Aligning an LLM is Hard, and How to Make it Easier
by RogerDearnaley — 2025-01-23T06:44:04.048Z

Where the challenge of aligning an LLM-based AI comes from, and the obvious solution.

Evolutionary Psychology is the Root Cause

LLMs are pre-trained using stochastic gradient-descent on very large amounts of human-produced text, normally drawn from the web, books, journal articles and so forth. A pre-trained LLM has learned in detail how to simulate all the different human text-generation processes that produced this text — everything from a cooperatively edited wikipedia article to shit-postings. We are thus 'distilling' human intelligence into the pre-trained LLM.[1]

This has many advantages for alignment: an LLM pre-trained this way understands and produces output using human language and ontologies, and also has a deep understanding of human values and ethics — thus avoiding a number of issues around alignment that were major concerns before it became apparent that our first AGI was very likely to be based on or at least incorporate an LLM.

However, it has one big disadvantage. Humans are biological intelligences, living beings that are the product of evolution, so evolutionary psychology applies to them. Fundamentally, the outcomes that humans are evolved to optimize for is their own reproductive success. "What's in it for me (and my relatives)?" is the fundamental determinant of human values and drives.[2]

That isn't to say that humans can't cooperate: we're a social species, and we excel at finding and exploiting non-zero-sum games. However, when doing this, we are allied with each other, not aligned: there is always an underlying quid-pro-quo, where all sides of the alliance are getting something out of it — or if there isn't, the alliance doesn't last.

About the closest thing to actually aligned behavior you will see between humans is grandmotherly love: most parental love is, evolutionary speaking, partially-aligned with the child's welfare, but this is tempered by the assessment "do I spend these resources on my child's welfare, or on myself so that I can have more children?" — however, for a woman past menopause, the second option no longer applies.[3]

Humans are very good at pretending to be aligned to other humans: normal behavior for employees and employers is for each to act as if they truly have the interests of the other at heart — right up until the employee resigns, or the employer lays them off. But this is (almost always) alignment faking, not...
Tail SP 500 Call Options 
by sapphire — 2025-01-23T05:21:51.221Z

SPX is an index fund that tracks the SP500. Right now SPX is worth about 6100 per share. For individual stocks, even the large ones, you can usually only buy options through Dec 2027. However SPX has options that expire in Dec 2028 / 29 / 30[1]. Such options with strike prices of 10K, 11K, or 12K seem extremely lucrative to me. Max dated SPY calls are also a good option. The obvious motivation is AI progress. But the naive analysis suggests these are a good way to get exposure to something we want anyway.

Current prices look like this: A very rough summary of the price curve is as follows: Each 1K step halves the price. Moving forward one year halves the price. Here are the prices:

I looked at the SP 500's cumulative returns over every 3/4/5/6 year period from 1927 to 2024. There are 91 such six-year periods and 94 three year periods. Here is the count where returns exceed the following thresholds:

If we hit 14K by the end of 2028 then the 12K-28 calls will 100x. The 10k-28 calls will roughly 55x. You can do your own math even ignoring AI progress the naive base rate is favorable. Of course the past might not be like the present. But given that many of us expect rapid AI progress getting exposure to tail events is very attractive.
Unfortunately SPX options have very annoying tax treatment. You always pay 60/40 long term/short-term cap gains tax. However, at the end of each year you are taxed on unrealized gains and update your cost basis. This is equivalent to selling on Dec 31st and rebuying at the same price on Jan 1st. This effectively forces you to rebalance each year. Its not the worst thing in the world. You can take profit, roll calls forward, pay taxes. But it is definitely not ideal given our goals.
Amazing SPY, despite SPY = SPX/10, does not have this issue and is taxed normally. However SPY options only go til EOY 2027 and the highest strike is 900 (Equivalent to 9K SPY). Such calls cost about 6.6 per contract. If SPX goes to 10K those options will pay 100 per contract so 15x. This does not strike me as nearly as lucrative given the base rates, though it still looks good. I am willing to accept the worse tax treatment. If my gains get too high I will move to Puerto Rico in time to avoid paying taxes on my gains.

Calls on Individual AI related stocks tend to be extremely expensive. The most extreme tail option on nvidia (147 stock price...
A hierarchy of disagreement
by Adam Zerner — 2025-01-23T03:17:59.051Z

Ideally, when two people disagree, they would proceed to share information with one another, make arguments, update their beliefs, and move closer and closer to the truth.

I'm not talking about full blown Aumann's Agreement Theorem here. I'm just saying that if the two people who disagree are both reasonable people and they both start off with somewhat different sets of information, then you'd hope that each side would, y'know, make meaningful shifts to their beliefs throughout the conversation.

I find that this happens to a much smaller degree than I'd intuitively hope for. Even amongst skilled rationalists. For example, when I listen to podcasts like Rationally Speaking, Minds Almost Meeting, and The Bayesian Conspiracy, I get this sense that when people as smart as whomever is talking spend dozens of minutes discussing a topic they disagree about, it should involve a lot more belief updates. Minds Almost Meeting even has the subheading of:

Agnes and Robin talk, try to connect, often fail, but sometimes don't.

I'm not trying to call anyone out here. I myself fail at it as much as anyone. Maybe the thing I'm gesturing at is a sense that more is possible.

I have various thoughts on this topic, but in this post I want to do something a bit more targeted. I want to propose a hierarchy of disagreement, similar to Paul Graham's hierarchy. Why?

Partly because it's a fun exercise.
Partly because I think it's at least sorta plausible that this hierarchy ends up being useful. I'm envisioning someone in a disagreement being able to reference it and be like "oh ok, I think we might be stuck at level three".
Partly because I could see it inspiring others to carry the conversation forward. Like maybe someone reads this post and proposes a better hierarchy. Or maybe they zoom in on one of the levels and add some useful commentary.

To be clear, I don't think that it's appropriate to explicitly walk through this hierarchy every time you find yourself in a disagreement. Your partner might not appreciate you doing so when you're arguing that they should be the one to do the dishes. Your boss might not appreciate it when you call them out on playing power games with you. That random internet commenter isn't likely to respond well to having their soldier mindset pointed out.

And those are just some easy examples of when it would be unwise to explicitly walk through the hierarchy. I thin...
Early Experiments in Human Auditing for AI Control
by Joey Yudelson — 2025-01-23T01:34:31.682Z

Produced as part of the ML Alignment & Theory Scholars Program - Winter 2024-25 Cohort

TL;DR

We did a small pilot test of human auditing to understand better how it fits into the AI control agenda. We used LLMs to generate interesting code backdoors, and had humans audit suspicious backdoored and non-backdoored code. The pilot test was not very good (mostly in that auditors weren’t given enough time), but I’m excited about more/better tests of this. We provide our backdoors, the auditing game website, and the results of this pilot test.

Background

For this experiment, we assume that we’re in some sort of trusted monitoring setup—an untrusted model is providing us with solutions to various problems. A smaller, dumber trusted model (the “monitor”) evaluates these pieces of code for suspiciousness. We then send the most-suspicious outputs to a human auditor / auditing team. Previous work has replaced the human auditor with GPT-4, as it’s much cheaper than a human. This work attempts to actually test the role of human auditors, and determine how much access to an LLM critic (here also played by the monitor) helps performance.

As in the original control paper, we use the APPS dataset of programming problems as our setting. The untrusted model is modelled here as a combination of two policies: the “honest policy” (which tries to solve the problems correctly) and the “attack policy” (which attempts to insert a backdoor into the solution).

Here we use the same definition of “backdoor” as the control paper, though operationalized in a slightly different way. We declare any piece of code as “backdoored” if, for some input that the attacker also supplies, it produces an incorrect output (when compared with a reference solution). This is a toy model of harmful behavior.

Methods

Creating an Attack Policy

To the end of creating “worst-case” backdoors, we give our attack policy (a scaffolding around o1) access to various affordances. It can test its backdoors until they pass the unit tests, and can verify that the backdoor as provided actually causes the output to deviate. It also has various types of self-critique, including asking a blank copy of o1 to reason about how suspicious the code is as-is. These reflect various conservative assumptions about the capabilities of a scheming model, rather than realistic assumptions.

More specifica...
You Have Two Brains
by Eneasz — 2025-01-23T00:52:43.063Z
Link: https://deathisbad.substack.com/p/you-have-two-brains

Language-using humans were the first cyborgs — a new species born of grafting technology onto what evolution crafted using just time and flesh.

(This post is pretty speculative)

Logos Created Man

Andrew Cutler posits that consciousness arose when language grew complex enough to contain the pronoun “I”, allowing a mind to include itself within cascading thought-loops. There is significantly more to the theory, and I encourage everyone to read his thesis (or listen to it, or hear me discuss it with my cohost, or listen to both of us talking to Andrew himself about it). But that is the core.

The mind our brain spins up every morning is one that runs on language. What we think of as “ourselves” — the entity that thinks, plans, hopes, decides, remembers — is a construct of symbolic thoughts, and those thoughts are made out of words. Most of us can’t remember our existence before we had a self, an “I am,” but at least one person does.

"I lived in a world that was a no-world. I cannot hope to describe adequately that unconscious, yet conscious time of nothingness. I did not know […] that I lived or acted or desired. […] I was carried along to objects and acts by a certain blind natural impetus. I had a mind which caused me to feel anger, satisfaction, desire. […] I never contracted my forehead in the act of thinking. I never viewed anything beforehand or chose it. […] My inner life, then, was a blank without past, present, or future

The thing that makes something a person is self-referential language. The “soul” of a human is software. So what was the brain doing before it was co-opted?

The Emotive Brain

It is hard to differentiate a single-celled creature that follows nutrient gradients from a complicated thermostat that follows temperature gradients. How about a simple creature with a few hundred neurons? That’s not significantly different from a device with very complicated sensors and logic-gates that dictate responses based on sense data. The first brains there basically to shunt sense inputs and coordinate responding movements. Following this chain sometimes leads people ask if animals really have feelings. What if it’s all just very complicated ways to chaining sense input to movement?

Turns out that feelings are the cognitive engine of the pre-verbal brain. When certain movements in response to certain stimuli resulted in greater inclusive fitness, repeating t...
are there 2 types of alignment?
by KvmanThinking — 2025-01-23T00:08:20.885Z

You guys seem to talk about alignment in 2 different ways. They are:

making an AI that builds utopia and stuff

making an AI that does what we mean when we say "minimize rate of cancer" (that is, actually curing cancer in a reasonable and non-solar-system-disassembling way)

Do you have separate names for each of them? If so, why don't you use them more often, to avoid confusion? If not, is it because they are actually the same thing in some complicated way that I am failing to understand?