trentmkelly/LessWrong-43k huggingface dataset:
Update on the Mysterious Trump Buyers on Polymarket
by Annapurna — 2024-11-04T19:22:06.540Z
Link: https://jorgevelez.substack.com/p/the-mysterious-trump-buyers-on-polymarket-2

I've written a follow-up post on the mysterious Trump buyers on Polymarket. While mainstream media has extensively covered this story, it has overlooked some critical details—most notably, that this trader's bet on Trump is closer to $75 million USDC, making it the largest election market wager to date. Regardless of the outcome, Theo is poised to go down in history as the most significant bettor in prediction markets.

Link to Post
[Intuitive self-models] 8. Rooting Out Free Will Intuitions
by Steven Byrnes — 2024-11-04T18:16:26.736Z

8.1 Post summary / Table of contents

This is the final post of the Intuitive Self-Models series.

One-paragraph tl;dr: This post is, in a sense, the flip side of Post 3. Post 3 centered around the suite of intuitions related to free will. What are these intuitions? How did these intuitions wind up in my brain, even when they have (I argue) precious little relation to real psychology or neuroscience? But Post 3 left a critical question unaddressed: If free-will-related intuitions are the wrong way to think about the everyday psychology of motivation—desires, urges, akrasia, willpower, self-control, and more—then what’s the right way to think about all those things? In this post, I offer a framework to fill that gap.

Slightly longer intro and summary: Back in Post 3, I argued that the way we conceptualize free will, agency, desires, and decisions in the “Conventional Intuitive Self-Model” (§3.2) bears little relation to what’s actually happening in the brain. For example, for reasons explained in detail in §3.3 (brief recap in §8.4.1 below), “true” desires are conceptualized in a way that makes them incompatible with having any upstream cause. And thus:

Suppose I tell you that you want food right now “merely” because your empty stomach is releasing ghrelin hormone, which is triggering ghrelin-sensing neurons in your hypothalamus, which in turn are manipulating your motivation system. It probably feels like this information comes somewhat at the expense of your sense of free will and agency. You don’t really want food right now—you’re just being puppeteered by the ghrelin! Right?

Next, suppose I tell you that your deepest and truest desires—your desires for honor and integrity, truth and justice—feel like the right thing to do “merely” because different neurons in your hypothalamus, neurons that are probably just a centimeter or two from those ghrelin-sensing ones above, are manipulating your motivation system in a fundamentally similar way.[1] It probably feels like this information comes deeply at the expense of your sense of free will and agency. You might even feel outraged and insulted!

And taking that even further still, suppose I inform you that absolutely everything you want and do is “merely” an output of your brain algorithm as a whole, which is a complex contraption inexorably cranking forward via the laws of physics and chemis

...
Option control
by Joe Carlsmith — 2024-11-04T17:54:03.073Z

Introduction and summary

(This is the third in a series of four posts about how we might solve the alignment problem. See the first post for an introduction to this project and a summary of the posts that have been released thus far.)

In the first post in this series, I distinguished between “motivation control” (trying to ensure that a superintelligent AI’s motivations have specific properties) and “option control” (trying to ensure that a superintelligent AI’s options have specific properties). My second post surveyed some of the key challenges and available approaches for motivation control. In this post, I do the same for option control.

Here’s a summary:

I start by briefly arguing that option control still matters even in the context of superintelligences.
I then distinguish between a few different types of option control – and in particular, between approaches that focus on limiting/eliminating paths to successful takeover (what I call “takeover-path restriction”) vs. approaches that focus on influencing an AI’s incentives in some other way (“other incentive shaping”).

Diagram of two approaches to option control

I then discuss takeover path restriction in some detail.

I start by drawing a few comparisons and contrasts with motivation control – in particular, takeover path restriction does not require the same kind of generalization; and sufficiently high quality oversight can turn motivation control issues into option control issues (because it can sufficiently restrict available takeover paths).
I then discuss what I see as the four key issues with takeover path restriction (in the context of superintelligence), namely:

Superhuman strategies: Superintelligences can use takeover strategies that humans couldn’t generate ahead of time or detect while they’re being pursued.
Opacity: again, it would help a lot to have direct epistemic access to internal variables like an AI’s capabilities, world model, reasoning, motivations, and so on; but in current ML models, you don’t.
Oversight scarcity/latency: Even if we had reliable oversight, it might be too expensive to apply to enough actions, and/or too slow to respond to bad behavior.
Adversarial dynamics: an AI might be actively optimizing against your efforts to restrict its takeover options, and the AIs you try to get to help you might sabotage your efforts as well.

The...
Noticing the World
by EvolutionByDesign — 2024-11-04T16:41:44.696Z

There are a lot of problems facing the world right now. To decide which to confront, you must know enough about the overall class of problems to say "yes, I want to work on x" because I feel that breakthrough "y" makes it possible to solve this problem and others aren't working on it for so and so reasons. How does one:

Notice the problems facing the world and their severity
Become aware of the new technologies are being developed right now
Recognize that "y" makes it possible to crack "x"?

I'd appreciate recommendations of books / textbooks / videos / articles / resources that helped you with any of the above (I can probably get access to any book you suggest - feel free to suggest literature that's out of print).

An example of this would be realizing that proteins ~= molecular nanotechnology (2), that molecular nanotechnology has such and such applications (1), and that with new breakthroughs in the field, such nanotech has now become a lot more possible than it was before (3).
The current state of RSPs
by Zach Stein-Perlman — 2024-11-04T16:00:42.630Z

This is a reference post. It contains no novel facts and almost no novel analysis.

The idea of responsible scaling policies is now over a year old. Anthropic, OpenAI, and DeepMind each have something like an RSP, and several other relevant companies have committed to publish RSPs by February.

The core of an RSP is a risk assessment plan plus a plan for safety practices as a function of risk assessment results. RSPs are appealing because safety practices should be a function of warning signs, and people who disagree about when warning signs are likely to appear may still be able to agree on appropriate responses to particular warning signs. And preparing to notice warning signs, and planning responses, is good to do in advance.

Unfortunately, even given which high-level capabilities are dangerous, it turns out that it's hard to design great tests for those capabilities in advance. And it's hard to determine what safety practices are necessary and sufficient to avert risks. So RSPs have high-level capability thresholds but those thresholds aren't operationalized. Nobody knows how to write an RSP that's not extremely conservative that passes the LeCun test:

the LeCun Test: Imagine another frontier AI developer adopts a copy of our RSP as binding policy and entrusts someone who thinks that AGI safety concerns are mostly bullshit to implement it. If the RSP is well-written, we should still be reassured that the developer will behave safely—or, at least, if they fail, we should be confident that they’ll fail in a very visible and accountable way.

Maybe third-party evaluation of models or auditing of an RSP and its implementation could help external observers notice if an AI company is behaving unsafely. Strong versions of this have not yet appeared.[1]

Anthropic

Responsible Scaling Policy

Basic structure: do evals for CBRN, AI R&D, and cyber capabilities at least every 6 months. Once evals show that a model might be above a CBRN capability threshold, implement the ASL-3 Deployment Standard and the ASL-3 Security Standard (or restrict deployment[2] or pause training, respectively, until doing so). Once evals show that a model might be above an AI R&D capability threshold, implement the ASL-3 Security Standard.

Footnotes removed and formatting edited:

Chemical, Biological, Radiological, and Nuclear (CBRN) weapons

The ability to significant

...
Does the "ancient wisdom" argument have any validity? If a particular teaching or tradition is old, to what extent does this make it more trustworthy?
by SpectrumDT — 2024-11-04T15:20:14.822Z

Proponents of spirituality and alternative medicine often use the argument "this has been practiced for 2000 years", with the subtext "therefore it must work". Does this argument have any validity?

At first glance I want to reject the argument entirely, but that might be premature. Are there situations where this kind of argument is valid or somewhat valid?

I was reminded of this question when I read Shaila Catherine's book The Jhanas (about certain ecstatic meditation states mentioned in Buddhism) and she said something like: "Trust in the method. Buddhists have been practicing it for 2600 years. It works. Your mind is not the exception." This argument did not seem valid to me, because AFAIK Buddhist monasteries do not publish records of how many of their monks achieve which states and insights - au contraire, I believe monks have a taboo against talking about their attainments. So I know of no evidence that most practitioners can achieve jhana. From what I know, it is entirely plausible that only a small fraction of practitioners ever succeed at these instructions, and that therefore their minds are the exception, not mine.
A brief history of the automated corporation
by owencb — 2024-11-04T14:35:04.906Z
Link: https://strangecities.substack.com/p/a-brief-history-of-the-automated

Looking back from 2041

When people in the early 21st Century imagined an AI-empowered economy, they tended to project person-like AI entities doing the work. “There will be demand for agent-like systems,” they argued, “so we’ll see AI labs making agents which can then be deployed to various problems”.

We now know that that isn’t how it played out. But what led to the largely automated corporations that we see today? Let’s revisit the history:

In the mid–late 2020s, as AI systems became reasonably reliable at many tasks, workers across the economy started consulting them more on an everyday basis

People and companies started more collection of data showing exactly what they wanted from different tasks

Systematizers and managers began building company workflows around the automation of tasks

They would build systems to get things into shapes known to work especially well for automation — in many cases using off-the-shelf software solutions — and direct more of the work into these routes
In many cases, the automation of a particular tasks involved brief invocation of specialized agent-like systems; but nothing like the long-term general purpose actors imagined in science fiction

As best practices emerged for automating taskflows, in the early–mid 2030s we saw the start of widespread automation of automation — people used specialized AI systems (or consultants relying on such systems) to advise on which parts of the workflow should be automated and how
For a while, human experts and managers kept a close eye on these automated loops, to catch and correct errors

But it wasn’t long before these management processes themselves were largely automatable (or redundant), and humans just stayed in loop for the high-level decisions about how to arrange different workflows and keep them integrated with the parts still done by humans
Although there are some great anecdotes of failures during that time, the broad trend was towards it being economically efficient to automate larger and larger swathes of work

At this stage, many companies were still run by people who were slow adopters of technology

Over the mid and late 2030s, many of these went out of business, as they failed to be competitive on price
There was significant social unhappiness at the shocks to the labour market

Lagging behind the automation of existing workflows was the automation of creating new workflows

Still, this

...
Abstractions are not Natural
by Alfred Harwood — 2024-11-04T11:10:09.023Z

(This was inspired by a conversation with Alex Altair and other fellows as part of the agent foundations fellowship, funded by the LTFF)

(Also: after I had essentially finished this piece, I was pointed toward the post Natural abstractions are observer-dependent which covers a lot of similar ground. I've decided to post this one anyway because it comes at things from a slightly different angle.)

Here is a nice summary statement of the Natural Abstractions Hypothesis (NAH):

The Natural Abstraction Hypothesis, proposed by John Wentworth, states that there exist abstractions (relatively low-dimensional summaries which capture information relevant for prediction) which are "natural" in the sense that we should expect a wide variety of cognitive systems to converge on using them.

I think that this is not true and that whenever cognitive systems converge on using the same abstractions this is almost entirely due to similarities present in the systems themselves, rather than any fact about the world being 'naturally abstractable'.

I tried to explain my view in a conversation and didn't do a very good job, so this is a second attempt.

To start, I'll attempt to answer the following question:

Suppose we had two cognitive systems which did not share the same abstractions. Under what circumstances would we consider this a refutation of the NAH?

Systems must have similar observational apparatus

Imagine two cognitive systems observing the the same view of the world with the following distinction: the first system receives its observations though a colour camera and the second system receives its observations through an otherwise identical black-and-white camera. Suppose the two systems have identical goals and we allow them both the same freedom to explore and interact with the world. After letting them do this for a while we quiz them both about their models of the world (either by asking them directly or through some interpretability techniques if the systems are neural nets). If we found that the first system had an abstraction for 'blue' but the second system did not, would we consider this evidence against the NAH? Or more specifically, would be consider this evidence that 'blue' or 'colour' were not 'natural' abstractions? Probably not, since it is obvious that the lack of a 'blue' abstraction in the second system comes from its observational apparatus, not from any feature ...
[Linkpost] Building Altruistic and Moral AI Agent with
Brain-inspired Affective Empathy Mechanisms
by Gunnar_Zarncke — 2024-11-04T10:15:35.550Z
Link: https://arxiv.org/pdf/2410.21882

Abstract

As AI closely interacts with human society, it is crucial to ensure that its decision-making is safe, altruistic, and aligned with human ethical and moral values. However, existing research on embedding ethical and moral considerations into AI remains insufficient, and previous external constraints based on principles and rules are inadequate to provide AI with longterm stability and generalization capabilities. In contrast, the intrinsic altruistic motivation based on empathy is more willing, spontaneous, and robust. Therefore, this paper is dedicated to autonomously driving intelligent agents to acquire morally behaviors through human-like affective empathy mechanisms. We draw inspiration from the neural mechanism of human brain’s moral intuitive decision-making, and simulate the mirror neuron system to construct a brain-inspired affective empathydriven altruistic decision-making model. Here, empathy directly impacts dopamine release to form intrinsic altruistic motivation. Based on the principle of moral utilitarianism, we design the moral reward function that integrates intrinsic empathy and extrinsic self-task goals. A comprehensive experimental scenario incorporating empathetic processes, personal objectives, and altruistic goals is developed. The proposed model enables the agent to make consistent moral decisions (prioritizing altruism) by balancing self-interest with the well-being of others. We further introduce inhibitory neurons to regulate different levels of empathy and verify the positive correlation between empathy levels and altruistic preferences, yielding conclusions consistent with findings from psychological behavioral experiments. This work provides a feasible solution for the development of ethical AI by leveraging the intrinsic human-like empathy mechanisms, and contributes to the harmonious coexistence between humans and AI. [emphasis mine]
Context-dependent consequentialism
by Jeremy Gillen — 2024-11-04T09:29:24.310Z

This dialogue is still in progress, but due to other commitments we don't have much time to continue it. We think the content is interesting, so we decided to publish it unfinished. We will maybe very slowly continue adding to it in the future, but can't commit to doing so.

Jeremy Gillen

Context: I wrote a shortform about conditioning on high capability AI, and the ways I think my past self messed that up. In particular, I claimed that conditioning on high capability means that the AI will be goal-directed.

Matt responded:

Distinguish two notions of "goal-directedness":

The system has a fixed goal that it capably works towards across all contexts.
The system is able to capably work towards goals, but which it does, if any, may depend on the context.

My sense is that a high level of capability implies (2) but not (1). And that (1) is way more obviously dangerous. Do you disagree?

I responded that (2) is unstable.

So here's a few claims that I think we disagree on that might be interesting to discuss:

High capability implies long-term goals that are stable across context changes.
Context dependent goals are unstable under reflection.

Does this context seem right? And where would you like to start?

mattmacdermott

Yeah, this seems like a good place to start.
You have an argument that conditioning on high capability implies a dangerous kind of goal-directedness, the kind where a system has a fixed goal that it optimises across all contexts, and where it's very likely to want to seize control of resources to act towards that goal.
This argument seems a bit shakey to me in some of the key places, so I'd like to see if you can fill in the details in a way I find convincing.
To start with, I think we both agree that high capability at performing certain tasks requires some amount of goal-directedness. For example, if my AI can complete a 6 month ML research project using the same sort of methods a human uses, it's going to need to be able to identify an objective, break it down into sub-objectives, plan towards them, test its ideas, and revise its ideas, plans and objectives over time.
The first potential disagreement point I want to check is: do you think it's physically possible for such a system to exist, without having a goal inside it that it wants to optimise at all costs? That is, can the system be a research-project-executing machine, rather than a send-the-world-i

...
Survival without dignity
by L Rudolf L — 2024-11-04T02:29:38.758Z
Link: https://nosetgauge.substack.com/p/survival-without-dignity

I open my eyes and find myself lying on a bed in a hospital room. I blink.

"Hello", says a middle-aged man with glasses, sitting on a chair by my bed. "You've been out for quite a long while."

"Oh no ... is it Friday already? I had that report due -"

"It's Thursday", the man says.

"Oh great", I say. "I still have time."

"Oh, you have all the time in the world", the man says, chuckling. "You were out for 21 years."

I burst out laughing, but then falter as the man just keeps looking at me. "You mean to tell me" - I stop to let out another laugh - "that it's 2045?"

"January 26th, 2045", the man says.

"I'm surprised, honestly, that you still have things like humans and hospitals", I say. "There were so many looming catastrophes in 2024. AI misalignment, all sorts of geopolitical tensions, climate change, the fertility crisis. Seems like it all got sorted, then?"

"Well", the man says. "Quite a lot has happened in the past 21 years. That's why they wanted me to talk to you first, before the doctors give you your final checkup." He offers his hand for me to shake. "My name is Anthony. What would you like to ask?"

"Okay, well, AI is the obvious place to start. In 2024, it seemed like we'd get human-level AI systems within a few years, and who knows what after that."

"Aah", Anthony says, leaning back in his chair. "Well, human-level, human-level, what a term. If I remember correctly, 2024 is when OpenAI released their o1 model?"

"Yes", I say.

"o1 achieved two notable things. First, it beat human subject-matter experts with PhDs on fiendishly-difficult and obscure multiple-choice science questions. Second, it was finally able to play tic-tac-toe against a human without losing. Human-level at both, indeed, but don't tell me you called it in advance that those two events would happen at the same time!"

"Okay, so what was the first important real-world thing they got superhuman at?"

"Relationships, broadly", Anthony says. "Turns out it's just a reinforcement learning problem: people interact and form personal connections with those that make them feel good."

"Now hold on. Humans are _good_ at human-to-human relationships. It's not like number theory, where there was zero ancestral environment incentive to be good at it. You should expect humans to be much better at relationships than most things, on some sort of objective scale."

"Sure, but also every human wants something from you, and ...