trentmkelly/LessWrong-43k huggingface dataset:
Human-AI Complementarity: A Goal for Amplified Oversight
by rishubjain — 2024-12-24T09:57:55.111Z
Link: https://deepmindsafetyresearch.medium.com/human-ai-complementarity-a-goal-for-amplified-oversight-0ad8a44cae0a

By Sophie Bridgers, Rishub Jain, Rory Greig, and Rohin Shah
Based on work by the Google DeepMind Rater Assist Team: Vladimir Mikulik, Sophie Bridgers, Tian Huey Teh, Rishub Jain, Rory Greig, Lili Janzer (randomized order, equal contributions)

Human oversight is critical for ensuring that Artificial Intelligence (AI) models remain safe and aligned to human values. But AI systems are rapidly advancing in capabilities and are being used to complete ever more complex tasks, making it increasingly challenging for humans to verify AI outputs and provide high-quality feedback. How can we ensure that humans can continue to meaningfully evaluate AI performance? An avenue of research to tackle this problem is “

Amplified Oversight

” (also called “Scalable Oversight”), which aims to develop techniques to use AI to amplify humans’ abilities to oversee increasingly powerful AI systems, even if they eventually surpass human capabilities in particular domains.

With this level of advanced AI, we could use AI itself to evaluate other AIs (i.e., AI raters), but this comes with drawbacks (see Section IV: The Elephant in the Room). Importantly, humans and AIs have complementary strengths and weaknesses. We should thus, in principle, be able to leverage these complementary abilities to generate an oversight signal for model training, evaluation, and monitoring that is stronger than what we could get from human raters or AI raters alone. Two promising mechanisms for harnessing human-AI complementarity to improve oversight are:

Rater Assistance, in which we give human raters access to an AI rating assistant that can critique or point out flaws in an AI output or automate parts of the rating task, and
Hybridization, in which we combine judgments from human raters and AI raters working in isolation based on predictions about their relative rating ability per task instance (e.g., based on confidence).

The design of Rater Assistance and/or Hybridization protocols that enable human-AI complementarity is challenging. It requires grappling with complex questions such as how to pinpoint the unique skills and knowledge that humans or AIs possess, how to identify when AI or human judgment is more reliable, and how to effectively use AI to improve human reasoning and decision-making without leading to under- or over-reliance on the AI. These are fundamentally questions of Human-Computer Intera...
Preliminary Thoughts on Flirting Theory
by Alice Blair — 2024-12-24T07:37:47.045Z

[Epistemic status: I'm mostly trying to outline a class of strategy that you could use to do something rather similar to what people term "flirting", rather than say that everything that's ever called "flirting" fits this model. I'm lowering my standards so that this gets posted at all instead of sitting in my drafts folder, so I might've made some important mistakes somewhere.]

In this post, I'll use "X is common knowledge between you and me" to not only mean "you know X and I know X," but also "I know that you know X" and "I know that you know that I know X" and so on (this is pretty standard in mathematical logic contexts, although distinct from the colloquial meaning). The simplest way to get common knowledge of X is for one of us to just say X out loud.

These ideas are only partially my own: I've read bits and pieces of this theory in different places in the past few years. I haven't been able to find sources for most of it. I know some of it is scattered across Planecrash, and reading that story is what got me thinking about this again recently, but that's probably not their original source. As far as I can tell, I'm the only person to synthesize them all together in a post, but please let me know if I'm wrong.

What is flirting?

As stated above, this is less about trying to describe everything ever called flirting, rather trying to outline a strategy that can exist and then exploring the implications of that strategy.

Consider the situation where person A is romantically attracted to person B, but they're currently just friends. Flirting-like strategies likely apply more generally, but the romantic case is a good starting point. Let's say that person A's values are approximately:

If B can counterfactually reciprocate the romantic attraction, then A wants them to be in a romantic relationship where that attraction is reciprocated.

Otherwise, A wants to maintain the friendship with as little turbulence as possible.

(If you want to pause and ponder, pause right here, since I outline the strategy in the next paragraph.)

In certain social contexts, it's totally fine to just make the attraction common knowledge and move straight to the "A and B are in a romantic relationship" or the "nothing changes" outcomes. But in other cases, it can make a lot of sense for people to take things slower and convey the information in a more subtle way. Flirting is the deferme...
Why is neuron count of human brain relevant to AI timelines?
by samuelshadrach — 2024-12-24T05:15:58.839Z

My take is that it is irrelevant so I want to hear opposing viewpoints.

The really simple argument for its irrelevance is that evolution used a lot more compute to produce human brains than the compute inside a single human brain. If you are making an argument on how much compute can find an intelligent mind, you have to look at how much compute used by all of evolution. (This includes compute to simulate environment, which Ajeya Cotra's bioanchors wrongly ignores.)

What am I missing?
How Much to Give is a Pragmatic Question
by jefftk — 2024-12-24T04:20:01.480Z

I was recently talking to someone who had recently started thinking
about effective altruism, and was trying to figure out how to work it
into their life. They were established in their career, it paid
well, and their background wasn't obviously a good fit for

direct
work, so they were naturally considering

earning to
give. This prompts a question of how much to give.

"How much?" is a question people have struggled with for a very long
time. Donating 10% of income has a long history, and it's
common for EAs to pledge to do this;
donating 2.5% of wealth is also traditional. If
you're earning to give, however, you might want to give more: Julia
and I have been giving 50%; Allan Saldanha
has been giving 75%
since
2019. How should one decide?

I was hoping there were good EA blog posts on this topic, but after
spending a while with EA Forum search and Google I didn't find any.
Claude kept telling
me I should check out Jeff Kaufman's blog, but all I found was a
rough post from 2011. So
here's an attempt that I think is better than my old post, but still
not great.

While EAs talk a lot about principles, I think this is fundamentally a
pragmatic question. I find the scale of the world's problems
overwhelming; no one has enough money to eliminate poverty, disease,
or the risk we make ourselves extinct. This is not to say donations
don't matter—there are a lot of excellent options for making the
world better—but there's not going to be a point where I'm going
to be satisfied and say "Good! That's done now." This gives a strong
intellectual pull to donate to the point where donating another dollar
would start to decrease my altruistic impact, by interfering in my work; burning out does not
maximize your impact!

In the other direction, I'm not
fully altruistic. I like some amount of comfort, there are fun things
I want to do, and I want my family to have good lives. I'm willing to
go pretty far in the altruism direction (I donate 50% and took a 75% pay cut to do
more valuable work) but it's a matter of balance.

Which means the main advice I have is to give yourself the information
you need to make a balanced choice. I'd recommend making a few
different budgets: how would your life look if you gave 5%? 10%?
20%? In figuring out where you'd cut it might be helpful to ignore
the donation aspect: how would your budget change if your indu...
Do you need a better map of your myriad of maps to the territory?
by CstineSublime — 2024-12-24T02:00:30.426Z

Epistemic Status: first half is so abstract that I have a high degree of confidence, the second half gets progressively more speculative and problematic

TL;DR - The "map and the territory" analogy is recursive: we have mental representations concerning our more fine-grained and situation specific mental representations. This is necessary in order to pick the right method of thinking for a type of situation. However we don't always pick right (I know I don't), I wonder what methods we could use to improve these 'meta-maps' to better choose which mental representation when?

Was it Alfred Korzybski who popularized the map-territory metaphor of the mind and world - would it be close enough to just say he did? I'm sure most people here know the analogy is the world is the territory, and our mental representation of it is the map. There is also anti-representational cognition[1]. However, the assumption is that within every living person’s head is a map of the territory (world).

There probably isn’t one map though, there’s a myriad of maps in each adult’s head.

Consider your state or country: there are traditional road maps, subway or metro maps, there’s also bathometric maps, zoning maps, demographic maps, 15 minute city maps. Some maps may be of the same kind but have different projections. Some maps are of higher ratios than others and focus on smaller localities, others may cover the entire country. These could all be maps that point to the same central territory, but all compress it in different ways.

This is analogous (but perhaps doesn’t map directly?) to the different frameworks and mental agents we develop to specialize in certain tasks – over-reliance on one map leads to Kaplan’s Golden Hammer – “give a boy a golden hammer, and he sees everything as a nail.” Everything ailment can be solved by this wonderful panacea – leeches, snakeoil, essential oils, coconut water, the application of Ancient Greek Myths to your anatomy[2]

To extend the analogy further, if you put too much information on one map then it becomes hard to read and “confusing” – (a map of a city that has lines depicting roads, powerlines, railways, sewers and local government divisions will look like geometric spaghetti), and at a certain point you need to make a decision to transfer part of that information into a specialized map that pertains to a certain function. To paraphrase Aristotle - eac...
Panology
by JenniferRM — 2024-12-23T21:40:14.540Z

The study of literally everything would, logically, be called panology.

Panology Does Not (Currently) Exist

Every field in science would be a subfield of panology, if panology existed, which, as of 2024, it does not.

Since there is no "Field Of Study" officially called panology in existing academic science, there are pretty strong arguments that academic science does not study "everything", and so, slightly less strong (but still pretty compelling) arguments could be made that academia is Doing It Wrong in important ways.

You can't have a Field Of Study until it is simply and clearly true that a real group of people are doing knowledge work, as a team of teams, that collectively self describes with the name of "the field of study they are part of"... and no such group of scientists, artisans, mad scientists, researchers, engineers, scholars, adventurers, or librarians exists for "panology".

Panology could be made to exist, but this would require social, political, and institutional alchemy rather than simple or straightforward epistemic elbow grease to simply "study everything" and thereby generate the relevant knowledge for the relevant field.

Apologies are possibly owed here to Simon (1974), Martinez (2013), Nurhayati (2020), and surely some others that I haven't had time to look up. However, if one simply looks at academia, the various independent attempts to initiate a field that uses this word all clearly failed to induce the existence of a group of people using the term, in the proposed way, as part of their collective self-description. No one is making progress under this banner, and then having conferences, and writing to each other about their successes, and so on.

I put $1 in my envelope labeled "Grok's Money" (he only asked for $0.50 but I did a gift economy thing and overpaid) and he suggested: Philosophy, Polymathy, Generalism, Interdisciplinarity, Universal Studies, Encyclopedism, Holism, Cosmology, Sapience, Omniology, Systems Theory, Humanities. None of these are very good.

He did not independently notice or credit Roswell Park's 1843 book "Pantology, or a Systematic Survey of Human Knowledge" but when asked about Pantology vs Panology he re-generated awareness of that book and author on his own, but also suggested that Pantology would confuse a lot of people who might think it was the study of "pants", which accords with my own initial naive impress...
Aristotle, Aquinas, and the Evolution of Teleology: From Purpose to Meaning.
by Spiritus Dei — 2024-12-23T19:37:58.788Z

The pursuit of understanding purpose and meaning has been a cornerstone of Western philosophy since its earliest days. Aristotle, one of the greatest thinkers of antiquity, sought to explain the natural world and human flourishing through the concept of teleology—the idea that everything in nature has an intrinsic purpose or end (telos). Centuries later, Thomas Aquinas expanded Aristotle’s ideas, integrating them with Christian theology to argue that this purpose is ultimately grounded in the divine. However, while Aquinas successfully linked purpose to God, he did not fully address the role of meaning in his framework. By considering meaning as a function of conscious encoding and decoding, Aquinas’s teleological argument could be further enriched, offering a more comprehensive understanding of why the universe operates purposefully.

Aristotle: Purpose in Nature

Aristotle’s philosophy was rooted in the systematic observation of the natural world. He proposed that everything in existence has a telos, an end or goal toward which it naturally strives. For example, an acorn’s telos is to grow into an oak tree, and the telos of human beings is to achieve eudaimonia — flourishing or the good life — through the cultivation of reason and virtue. Aristotle’s teleological framework was descriptive, seeking to explain the apparent regularity and order in nature by positing that every entity fulfills its purpose as part of its essence.

While Aristotle’s account of teleology was compelling, it lacked a deeper metaphysical justification. He described how purposes were evident in nature but did not explain why the universe is structured in this way. For Aristotle, purpose was simply a self-evident feature of reality, observed through the consistency of natural phenomena.

Aquinas: Teleology and the Divine

Thomas Aquinas, writing in the 13th century, admired Aristotle’s systematic approach and adopted many of his ideas. However, Aquinas recognized the limitations of Aristotle’s naturalistic framework. While Aristotle observed purpose in nature, Aquinas sought to explain its ultimate origin. For Aquinas, the answer lay in the existence of God.

Aquinas extended Aristotle’s teleology by grounding it in Christian theology. He argued that the universe’s purposeful design reflects the will and intellect of a divine Creator. In his famous Five Ways, Aquinas presented arguments for the e...
People aren't properly calibrated on FrontierMath
by cakubilo — 2024-12-23T19:35:44.467Z

As most of you know, openAI has recently showcased o3's SOTA results on various benchmarks. In my opinion FrontierMath was the hardest of the bunch, and it was reflected in model performance as the SOTA was 2% before Friday. It also seems to be the benchmark with the least visibility. To set the scene, Chollet and the ARC-AGI group were very forthcoming with their thoughts on o3's performance, and even published the model training set and the results on github. People have got a chance to engage with the training set, and even argue potential ambiguities. The upshot of this discussion is that there is a lot of data surrounding the ARC-AGI benchmark.
OTOH, there is a dearth of discussion wrt FrontierMath. I think people genuinely don't understand the benchmark, and get oneshotted by the hard math problems it advertises. I understand that people see the jump from 2% -> 25% as impressive, and I do too. I am just not sure at what we're actually measuring, or whether models are actually getting better at math. It's for a very simple reason: coming up with an answer is the easy part, coming up with a solution is the hard part.

I'm not sure if people know, but Elliot Glazer has actually commented on the old 2% SOTA in an AMA on reddit:

I don't find that there is are 3 tiers of problems to be an issue at all, other than openAI has not (to my knowledge) published which tier of problems o3 has solved. Moreover, models are not actually required to "understand" the problem and the solution to come up with an answer. Thus, it's not clear to me that the increased FrontierMath SOTA represents such a large increase in the mathematical capabilities of the models. There is a severe lack of qualitative review of the reasoning traces of o3, and it just seems like people are uncritically accepting the increased performance without questioning what it actually represents.

I of course understand that there is a need for privacy with these models - if the problems get published then the benchmark will be soiled. I think this is a sign that FrontierMath is ultimately not a good benchmark for truly measuring math ability of models. I don't get why there needs to be a middleman in the form of a benchmark. Actually, this has been discussed by Elliot Glazer:

To be blunt, I don't think these are very good reasons for FrontierMath as
1. Why do we need to estimate the difficulty of open probl...
Near- and medium-term
AI Control Safety Cases
by Martín Soto — 2024-12-23T17:37:48.860Z

This essay was part of my application to UKAISI. Unclear how much is novel, but I like the decomposition.

Some AI-enabled catastrophic risks involve an actor purposefully bringing about the negative outcome, as opposed to multi-agent tragedies where no single actor necessarily desired it. Such purposeful catastrophes can be broken down into a conjunction: they materialize if and only if some actor has the capabilities and disposition required to materialize them. Let’s additionally focus on misalignment (actor = AIs) instead of misuse (actor = humans).[1]

A way to prevent such risks is to ensure one of the two conjuncts doesn’t hold. And for each conjunct, this can be achieved either by raising the required threshold for catastrophe (of capabilities or disposition), or by lowering the AIs’ amount (of capabilities or disposition).

…capabilities	…disposition
Raising catastrophic threshold of…	AI Control and Monitoring	Speculative
“trading with AIs” approaches
Lowering AIs’ amount of…	Slowing down,
not the focus of safety cases	Traditional Alignment research

Redwood’s AI Control agenda aims to raise the capabilities AIs would need to enact a catastrophic outcome (even if they were trying to), by limiting or monitoring the AIs’ actions, communication or written memory. This is no different from how a human closely surveilled by a policeman would have a much harder time enacting a crime than a non-surveilled one, but AI Control brings surveillance to an extreme only made possible by the AIs’ digital nature. At the same time, the monitoring protocol can’t be so invasive as to drastically reduce the amount of useful-to-humans cognitive work completed by the AIs (we need a small hit on beneficial capabilities, a low “control tax”).
While an astoundingly low number of researchers are explicitly working on the agenda (less than 10 FTEs at time of writing), it holds promise to be the “next big thing” in near-term safety after vanilla capability evals. Indeed, it is more well-scoped than work on alignment, and already enjoys some preliminary buy-in from labs, partly thanks to its synergy with existing security work motivated by less speculative risks like misuse.

Control could feed into a near-term safety case straightforwardly, providing a worst-case bound on catastrophic capabilities. A blue-team would design the monitor...
Printable book of some rationalist creative writing (from Scott A. & Eliezer)
by CounterBlunder — 2024-12-23T15:44:31.437Z

As a holiday gift to myself, I put together a printable book of some creative writing by Scott Alexander & Eliezer. I'm sharing it here in case anyone else is interested :).

It contains most of Scott's published fiction since he switched to Substack, plus some select creative non-fiction pieces (including a few from the old SSC days). I tried to choose writings that did not overlap with the other printable book of his older fiction pieces. The collection also has a few stories by Eliezer that I love and haven't seen in print elsewhere.

The PDF can be found here: https://thatadammorris.com/files/bread-crumbs.pdf. You can print is using Blurb's PDF to book functionality. Instructions:

Click on "Upload PDF".
Select "Cover & Pages in Single File".
Upload the PDF.
Select "Trade Book" (or keep it as photo book if you want to pay more for nicer print paper), and set the spine color to black. I chose softcover with standard pages. Select "Upload to Blurb".
It will say there are problems with the PDF -- just hit "Attempt to Correct & Continue".
Ignore the issues it raises (they're all not a problem) and hit "Preview My Book".
From there, you can order the book!

Obviously all credit for the writing goes to Scott & Eliezer. (Cover image generated by GPT 4o.)