trentmkelly/LessWrong-43k huggingface dataset:
Is Optimization Correct?
by Yoshinori Okamoto — 2024-12-12T10:27:39.098Z

1. Is Optimization Correct?

The risk of "optimization" has long been recognized in AI alignment, such as the problems of instrumental convergence (Bostrom 2014). Nevertheless, it is difficult for AI designers to escape from the engineering concept of "optimization" because the concept of "optimization" is so strongly rooted in engineering design including AI design.

In value alignment, AI can be designed to optimize a certain value. Such AI alignment may seem harmless from intuition especially when the value is universally accepted as a good value, such as wellbeing, truth, justice, etc. However, such intuition may be wrong regarding advanced Artificial General Intelligence (AGI).

This article introduces the “Optimization Prohibition Theorem” as a concept that AI designers can refer, as an easy-to-understand design guideline of AGI (Okamoto 2024).

The Optimization Prohibition Theorem is a theorem that prohibits optimization targets based on engineering design principles in AI alignment of advanced AGI. The Optimization Prohibition Theorem is proven, under certain assumptions, as follows:

(Proof)

(1). (Assumption 1) AI alignment is done so that a plurality of AIs have optimization targets based on engineering design principles.

(2). (Assumption 2) A plurality of AIs are powerful AGIs that have sufficient resources and can achieve optimization goals using all means.

(3). (Assumption 3) The optimization targets for a plurality of AGIs are different and are not satisfied simultaneously.

(4). Under the above assumptions, if AGI1 and AGI2 try to achieve different optimization targets, only AGI2 is the obstacle of AGI1’s optimization. Both AGI1 and AGI2 have sufficient resources to achieve the optimization targets and they use any means to achieve the optimization targets, which will result in conflicts between AGIs, which damage AGIs and infringe human rights by collateral damages.

(5). Therefore, the Optimization Prohibition Theorem is established as a guideline for the design of advanced AGIs in AI alignment. (End of Proof)

As described above, according to the Optimization Prohibition Theorem, the optimization based on the engineering design principles can cause problems under certain conditions in the context of AI alignment of advanced AGIs.

2. For Future Discussions

This article introduced the concept of the "Optimization Prohibition The...
AXRP Episode 38.3 - Erik Jenner on Learned Look-Ahead
by DanielFilan — 2024-12-12T05:40:06.835Z

YouTube link

Lots of people in the AI safety space worry about models being able to make deliberate, multi-step plans. But can we already see this in existing neural nets? In this episode, I talk with Erik Jenner about his work looking at internal look-ahead within chess-playing neural networks.

Topics we discuss:

How chess neural nets look into the future

The dataset and basic methodology

Testing for branching futures?

Which experiments demonstrate what

How the ablation experiments work

Effect sizes

X-risk relevance

Follow-up work

How much planning does the network do?

Daniel Filan (00:09):
Hello, everyone. This is one of a series of short interviews that I’ve been conducting at the Bay Area Alignment Workshop, which is run by FAR.AI. Links to what we’re discussing, as usual, are in the description. A transcript is, as usual, available at axrp.net. And, as usual, if you want to support the podcast, you can do so at patreon.com/axrpodcast. Well, let’s continue to the interview.

(00:28):
All right. Well, today, I’m speaking with Erik Jenner. Erik, can you say a few words about yourself?

Erik Jenner (00:35):
Yeah. I’m currently a third-year PhD student at UC Berkeley at the Center for Human-Compatible AI, working on various things around model internals there.

Daniel Filan (00:45):
Cool. And right now, we’re at this alignment workshop being run by FAR.AI. How are you finding it?

Erik Jenner (00:51):
It’s been fun so far. We’ve only had a few talks, but I thought all of them were interesting.

Daniel Filan (00:55):
Cool, cool.

How chess neural nets look into the future

Daniel Filan (00:57):
So speaking of work that you’ve done, I guess we’re going to talk about this chess paper that you’ve worked on. So that listeners can look it up, what’s the name of this paper?

Erik Jenner (01:08):
It’s called “Evidence of Learned Look-Ahead in a Chess-Playing Neural Network”.

Daniel Filan (01:11):
Okay, that sort of tells you what it is, but can you elaborate a little bit?

Erik Jenner (01:18):
Yeah, so I guess the question we’re asking ourselves is: neural networks are pretty good at playing chess now, and playing chess in the sense not just of having Monte Carlo tree search with a big explicit search tree, but also playing chess if you only give them one forward pass to make every single move.

(01:32):
And so the question is: how are they so good at playing che...
Public computers can make addictive tools safe
by dkl9 — 2024-12-11T19:55:22.818Z
Link: https://dkl9.net/essays/safe_addictives.html

Intermittently for the past four years, I read LessWrong and its ilk to hazardous excess. By teaching technical topics sans pressure to rigourously understand, heavy with microhumour and satisfying insights, it got my susceptible mind addicted. Intermittently for the past year, I also posted on LessWrong, with variable (mostly good) results.

For a couple years, I alternated between a couple computers, only one of which I fully controlled. I blocked the addictive nerdy blogs on the machine I controlled. I cross-posted and over-read on the others. Cold-turkeying off the secondary computers gets me to quit the gratuitous reading, but locks me off from posting for much of my potential audience. We can do better.

As I write this, I live a walkable, but tedious, distance from public computers. Public computers can conveniently access the web, with little restriction on legal content. My main device, complete with its careful configuration, is at home. Thus I sometimes access LessWrong, but such access mutually excludes the comfort of home, and so is only occasional. By only going to public computers shortly before nearby scheduled events I'd go to anyway, my time on them is urgently bounded.

This sparse, planned-out nature of my LessWrong usage leads me to cross-post writing and do little else on it. This is the best of both worlds. The strategy generalises to any addictive website with some legitimate uses: visit it rarely and deliberately, and you will cram in the good parts, leaving the distracting parts behind.
Solving Newcomb's Paradox In Real Life
by Alice Wanderland — 2024-12-11T19:48:44.486Z
Link: https://open.substack.com/pub/aliceandbobinwanderland/p/solving-newcombs-paradox-with-ai?r=24ue7l&utm_campaign=post&utm_medium=web

I noticed we now live in a world where we can run Newcomb's problem as an actual experiment, so I ran the experiment!

Roleplaying as the predictor in the Newcomb problem (with an LLM as the "decision maker") finally made me grok why one-boxing is the correct solution to the original problem.

Wondering if anyone else feels the same way?
The "Think It Faster" Exercise
by Raemon — 2024-12-11T19:14:10.427Z

Note: there is a shorter "Think it Faster" worksheet. I'm curious which one people find easier as an initial read.

Ultimately, I don’t want to solve complex problems via laborious, complex thinking, if I can help it. Ideally, I'd want to basically intuitively follow the right path to the answer quickly, with barely any effort at all.

For a few months I've been experimenting with the "How Could I have Thought That Thought Faster?" concept, originally described in a twitter thread by Eliezer:

Sarah Constantin: I really liked this example of an introspective process, in this case about the "life problem" of scheduling dates and later canceling them: malcolmocean.com/2021/08/int…

Eliezer Yudkowsky: See, if I'd noticed myself doing anything remotely like that, I'd go back, figure out which steps of thought were actually performing intrinsically necessary cognitive work, and then retrain myself to perform only those steps over the course of 30 seconds.

SC: if you have done anything REMOTELY like training yourself to do it in 30 seconds, then you are radically smarter/more able/etc than me and all the other people who do slower introspective practices.

SC: I don't know whether to be impressed or to roll to disbelieve.

EY: I mean I suspect that this actually requires something like a fast perceptual view of minds as engines and thoughts as doing work and like actually draws on my mind design knowledge, but, even so, I ask: Do you constantly look back and ask "How could I have thought that faster?"

SC: No, I've never asked that.

EY: Okay, well, every time I'm surprised by reality I look back and think "What about my model and my way of thinking could I change that would have predicted that better, without predicting a bunch of other things worse?"

I've been working to operationalize this as an exercise[1] you can train repeatedly, rather than hoping to remember to do when reality hands you a surprise. You can do the exercise after any difficult cognitive task (either a toy puzzle exercise, or a day-job project that took a long while).

What would have been necessary for you to just look at the situation, and automagically find the right solution? (without overfitting, or generalizing in a way that would cause you to think unproductive thoughts in other sorts of situations?)

The goal of this exercise is to identify:

skills you can train
principles you can apply
actio

...
Forecast With GiveWell
by ChristianWilliams — 2024-12-11T17:52:32.293Z
Link: https://www.metaculus.com/tournament/forecast-with-givewell/?

Update: The Prize Pool has increased to $3,500!

Thank you to donors Anton Makiievskyi, Ryan Kidd, @Austin, and @Saul Munn, who contributed via Manifund.

A total prize pool of $3,500 will be awarded for comments on eligible questions made before January 1, 2025. Get started here.

Ten prizes will be awarded as follows:

1st place: $700
2nd place: $600
3rd place: $500
4th place: $400
5th place: $300
6th place through 10th place: $200 each

Announcement Post

GiveWell’s mission is to find the charities that deliver the greatest impact per dollar spent to help donors decide where to give. To support this, they publish forecasts on expected grant outcomes. Now, Metaculus and GiveWell invite you to forecast and share your reasoning to help GiveWell help others.

Get started and provide feedback on GiveWell's funding decisions. $1,000 commenting prize pool. Deadline is December 1st, 2024.

How much will GiveWell update its cost-effectiveness estimate of GiveDirectly based on the results of the 2023-2024 follow-up of the Cash Transfers Study by UC Berkeley?
Will the study in Ghana find that RTS,S combined with malaria chemoprevention reduces clinical malaria more than RTS,S alone in the following age groups?
Will the study of supplying eyeglasses to near-vision impaired adults in Kenya and India find a boost to household consumption of the following levels?
Will the World Health Organization prequalify moxidectin before 2027?
Will GiveWell recommend grants of more than $5 million to support delivering chlorine and/or oral rehydration solution and zinc along with vaccine outreach services?
Will the study of oral rehydration solution and zinc distribution in Sierra Leone find a statistically significant increase in ORS/zinc usage by the recipients?
Will the study of chlorine distribution in Sierra Leone find a statistically significant increase in chlorine usage by the recipients?
How many chlorination grants will GiveWell recommend after September 20, 2024, and before January 1, 2026?
Will the household water chlorination rate be 65% or higher for households served in the first two years of Evidence Action's program in Madhya Pradesh and Andhra Pradesh?
Will in-line chlorination devices serve at least 6.7 million people in Madhya Pradesh at the end of the 4th year of Evidence Action's program?
Will in-line chlorination devices serve at least 5.8 million people in Andhra Pradesh

...
A shortcoming of concrete demonstrations as AGI risk advocacy
by Steven Byrnes — 2024-12-11T16:48:41.602Z

Given any particular concrete demonstration of an AI algorithm doing seemingly-bad-thing X, a knowledgeable AGI optimist can look closely at the code, training data, etc., and say:

“Well of course, it’s obvious that the AI algorithm would do X under these circumstances. Duh. Why am I supposed to find that scary?”

And yes, it is true that, if you have enough of a knack for reasoning about algorithms, then you will never ever be surprised by any demonstration of any behavior from any algorithm. Algorithms ultimately just follow their source code.

(Indeed, even if you don’t have much of a knack for algorithms, such that you might not have correctly predicted what the algorithm did in advance, it will nevertheless feel obvious in hindsight!)

From the AGI optimist’s perspective: If I wasn’t scared of AGI extinction before, and nothing surprising has happened, then I won’t feel like I should change my beliefs. So, this is a general problem with concrete demonstrations as AGI risk advocacy:

“Did something terribly bad actually happen, like people were killed?”
“Well, no…”
“Did some algorithm do the exact thing that one would expect it to do, based on squinting at the source code and carefully reasoning about its consequences?”
“Well, yes…”
“OK then! So you’re telling me: Nothing bad happened, and nothing surprising happened. So why should I change my attitude?”

I already see people deploying this kind of argument, and I expect it to continue into future demos, independent of whether or not the demo is actually being used to make a valid point.

I think a good response from the AGI pessimist would be something like:

I claim that there’s a valid, robust argument that AGI extinction is a big risk. And I claim that different people disagree with this argument for different reasons:

Some are over-optimistic based on mistaken assumptions about the behavior of algorithms;
Some are over-optimistic based on mistaken assumptions about the behavior of humans;
Some are over-optimistic based on mistaken assumptions about the behavior of human institutions;
Many are just not thinking rigorously about this topic and putting all the pieces together; etc.

If you personally are highly skilled and proactive at reasoning about the behavior of algorithms, then that’s great, and you can pat yourself on the back for learning nothing whatsoever from this particular demo—assuming that’s not just your hindsi

...
Why Isn't Tesla Level 3?
by jefftk — 2024-12-11T14:50:01.159Z

Many people who've used Tesla's "Full Self Driving" software are
pretty

excited
about it:

I can step into a Tesla today, press a destination, and go there
without touching the wheel or pedals. Sure it won't be flawless but
the fact is, I can. I can't do the same in any other consumer car, and
the closest thing is a Waymo. The effort is there, I think its just a
matter of time before we start seeing the legal stuff play out.

I think this is mostly not a legal issue. Let's take perhaps the most
favorable conditions for driverless cars:

A specific highway the manufacturer has checked

Heavy traffic, under 40mph

No rain or snow

No work zones

Daytime

Tesla could demonstrate their system worked sufficiently reliably in
these conditions that the person in the driver's seat could safely
work, read, or watch a movie, and Tesla could take full legal
responsibility for any crash. We know Tesla could legally do this
because (despite what the Secretary
of Transportation thinks) Mercedes already does, with Drive
Pilot, which they launched in Germany in 2022
and the US in 2023.

Then Tesla could gradually remove restrictions, as they were able to
demonstrate that we could trust their system with more complex
scenarios.

Tesla fans will often claim that Tesla could easily do
this ("FSD is practically perfected, with no accidents whatsoever,
under such conditions already"), but I don't think it's so clear.
This is a situation where getting to "impressive" levels of operation
is quite doable (ex: here's Cruise seven years ago
on in SF, dealing with many unusual situations) but getting to
"reliable enough that you don't have to supervise it" has been
incredibly hard (ex: Cruise is shutting
down, after dragging
a pedestrian under a car last year). And unlike most of their
competitors (including Mercedes) the Tesla vehicles don't have LIDAR,
which makes it even harder to get to that level of reliability.

Tesla has been making bold promises here since at least 2016,
when they claimed that Self-Driving was limited only by "extensive
software validation and regulatory approval":

Full Self-Driving Capability

Build upon Enhanced Autopilot and order Full Self-Driving Capability
on your Tesla. ... Please note that Self-Driving functionality is
dependent upon extensive software validation and regulatory approval,
which may vary widely b

...
Investing in Robust Safety Mechanisms is critical for reducing Systemic Risks
by Tom DAVID — 2024-12-11T13:37:24.177Z

This short paper was written quickly, within a single day, and is not highly detailed or fully developed. We welcome any feedback or suggestions you may have to improve or expand upon the ideas presented.

ABSTRACT
In this position paper, we argue that research on robustness against malicious instructions should be a core component of the portfolio of AI systemic risk mitigation strategies. We present the main argument raised against this position (i.e. the ease of safeguard tampering) and address it by showing that state-of-the-art research on tampering resistance offers promising solutions to make safeguard tampering costlier for attackers

1. INTRODUCTION

At the risk of sounding trivial, we assert that the first rampart against systemic risks from misuse of AI is the inability to elicit dangerous capabilities out of an advanced AI system. This implies:

Absence of harmful knowledge: If the AI does not know the harmful information (e.g., CRBN or cyber knowledge), then it cannot provide it.
Existence of robust safety mechanisms against malicious instructions: If the AI always refuses to provide potentially harmful information, then it will not provide it either.

Making progress on either of these conditions (and ideally both) is critical for reducing systemic risks from AI.
However, achieving the first condition is limited by significant technical challenges, including inadvertently reducing beneficial capabilities and potential reacquisition of dangerous knowledge. The feasibility of the second condition has been contested due to the ease of tampering with safeguards.

Ease of safeguard tampering argument:

It is possible to tamper with safeguards through fine-tuning or targeted model weight modification (e.g., refusal feature ablation).
Therefore, it is argued that focusing on making such safeguards robust against malicious instructions is futile.

We argue that this reasoning is flawed and that improving the robustness of safety mechanisms against tampering is essential.

2. SAFEGUARD TAMPERING EXISTS

Safeguard tampering is well-documented:

Fine-tuning has been proven effective in bypassing safeguards with minimal examples.
Mechanistic interpretability studies show safety refusal mechanisms can be bypassed using techniques like weights manipulation (e.g., abliteration).

These methods highlight the need for improved resistance to safeguard tampering.

3. EXAMINING THE LITERATURE O

...
Post-Quantum Investing: Dump Crypto for Index Funds and Real Estate?
by G — 2024-12-11T11:59:11.062Z

I'll keep this short.

Google’s Willow quantum chip significantly outpaces current supercomputers, solving tasks in minutes that would otherwise take billions of years.
Hypothesis: Accelerating advancements in tech and AI could lead to quantum supremacy arriving sooner than the 2030s, contrary to expert predictions.
Legacy banking systems, being centralized, could transition faster to post-quantum-safe encryption by freezing transfers, re-checking processes, and migrating to new protocols in a controlled manner.
Decentralized cryptocurrencies face bigger challenges:

Hard forks are difficult to coordinate across a decentralized network.
Transitioning to quantum-safe algorithms could lead to longer transaction signatures and significantly higher fees, eroding trust in the system.

If quantum computers compromise current cryptography, tangible assets (e.g., real estate, stock indices) may retain more value compared to digital assets like crypto.

Thoughts?
Low-effort review of "AI For Humanity"
by Charlie Steiner — 2024-12-11T09:54:42.871Z

Stumbled across a book in the new section of the library: "AI For Humanity," by Andeed Ma, James Ong (founder of the think tank AIII, which is also the sound I make when thinking about AI risk), and Siok Siok Tan. It's a mass-market-ish book about, well, AI for humanity, by a sampler of Singapore-centered technologists.

Chapter-by-chapter summary:

Different people have different opinions along both axes of "AI Impactful?" and "AI Risky?" We'll try to take them all seriously. Here's the example of Geoff Hinton talking about AI risk.
The nearest/most important 'AI Trap' is misinformation and general destruction of the epistemic environment.
Covers some sort of history of AI, with an emphasis on how vague the category 'AI' is and how functionally equivalent technology merely went "in camouflage" during putative AI Winters.
Meanderingly talks about approx. three "dilemmas": "Do we use AI at all?" "How to deal with the epistemic AI Trap?" and "How to align AI to human values [as yet unclear if this really means all human values or "fairness and transparency" sort of stuff] even though it's 'organic' [-ly adapting and improving]."
Four principles are supposed to deal with everything: Humans should be in control, AI should be contained, the human-AI relationship should be symbiotic, and humans should nurture the AI. A mixture of empty-feeling buzzword-based reasoning and non-empty discussion of historical chatbots ensues.
Starts talking about AGI, introduces the new buzzword "Sustainable AI" with three pillars: "governing AI to align with human values," "combine human desirability, technological feasibility, and business viability to weather AI boom-and-bust cycles," and "delivering on the greater good of sustainability and impact, such as the UN SDGs [United Nations sustainable development goals, e.g. ensuring clean water, affordable clean energy]." This section (pp 131-134) has some actual hot takes:

"Governance" (unclear if this means governments or internal corporate policy) should mandate AI that's aligned with humans and AI that helps predict and manage threats from AI.
Just fuse with the AI lol.
There should be massive impact-based funding of public-private partnerships using AI for the benefit of humankind.
...before whipping back to corporate-speak to say IBM could have done better with its foray into healthcare AI by using their framework. But surprise again, becaus

...