trentmkelly/LessWrong-43k huggingface dataset:
Sleeping Beauty – the Death Hypothesis
by Guillaume Charrier — 2023-05-05T23:32:20.278Z

Introduction

This post is intended as a contribution to the (already admittedly very large) corpus of discussion surrounding the Sleeping Beauty paradox. The statement of the paradox (which I will simply refer to here as the Paradox) itself is assumed to be known, as it is widely available on the internet, including on this site.

When I first learned of the Paradox, I was an intuitive one halfer. This post is meant to describe the reasoning path that led me to becoming a one thirder.

Specifically, for reasons I will further detail below, I believe that much of the confusion on the resolution of the Paradox stems from a failure to differentiate between two clearly distinct questions:

Q1: The experiment has started, and Sleeping Beauty is woken up. What is the probability that the coin landed on heads?

Q2: The experiment has started, and Sleeping Beauty is woken up. As a rational individual, what value should she propose for the probability that the coin landed on heads?

I believe that the answer to Q1 is one half, and that the answer to Q2 is one third. Since the question posed in the Paradox is Q2, I am a one thirder.

Let me try to explain why.

Step 1: Preliminary Remarks

In certain situations, the fact that the experimental observer is capable of proposing a probability value for a certain event matters to the probability value she should rationally propose for that event. For instance, let us consider a situation where Sleeping Beauty is subjected to the following experiment:

A fair coin is tossed.
Sleeping Beauty lies down on a mattress, for pillow she will rest her head on a fairly sizeable pack of fairly powerful explosives.
Sleeping Beauty is put to sleep.
If the coin landed on heads, Sleeping Beauty is woken up. However, one attosecond after she is woken up the pack of explosives is detonated and Sleeping Beauty dies. Because the pack was detonated so soon after she woke up, the detonation, and Sleeping Beauty’s death, occurred well before any thought or concept (which requires neuronal activity, which is comparatively slow), be it that of a single, simple numerical value can be formed in her brain, much less communicated to any third party.
If the coin landed on tails, Sleeping Beauty is woken up and then proposes, to the best of her rational abilities, a value for the probability that the coin landed on heads.

Note, that similar to conditions in the Paradox,...
Orthogonal's Formal-Goal Alignment theory of change
by Tamsin Leake — 2023-05-05T22:36:14.883Z
Link: https://carado.moe/formal-alignment-theory-change.html

We recently announced [Orthogonal, an agent foundations alignment research organization. In this post, I give a thorough explanation of the formal-goal alignment framework, the motivation behind it, and the theory of change it fits in.

The overall shape of what we're doing is:

Building a formal goal which would lead to good worlds when pursued — our best candidate for this is QACI

Designing an AI which takes as input a formal goal, and returns actions which pursue that goal in the distribution of worlds we likely inhabit

Backchaining: aiming at solutions

One core aspect of our theory of change is backchaining: come up with an at least remotely plausible story for how the world is saved from AI doom, and try to think about how to get there. This avoids spending lots of time getting confused about concepts that are confusing because they were the wrong thing to think about all along, such as "what is the shape of human values?" or "what does GPT4 want?" — our intent is to study things that fit together to form a full plan for saving the world.

Alignment engineering and agent foundations

Alignment is not just not the default, it's a very narrow target. As a result, there are many bits of non-obvious work which need to be done. Alignment isn't just finding the right weight to sign-flip to get the AI to switch from evil to good; it is the hard work of putting together something which coherently and robustly points in a direction we like.

as yudkowsky puts it:

The idea with agent foundations, which I guess hasn't successfully been communicated to this day, was finding a coherent target to try to get into the system by any means (potentially including DL ones).

Agent foundations/formal-goal alignment is not fundamentally about doing math or being theoretical or thinking abstractly or proving things. Agent foundations/formal-goal alignment is about building a coherent target which is fully made of math — not of human words with unspecified meaning — and figuring out a way to make that target maximized by AI. Formal-goal alignment is about building a fully formalized goal, not about going about things in a "formal" manner.

Current AI technologies are not strong agents pursuing a coherent goal (SGCA). The reason for this is not because this kind of technology is impossible or too confusing to build, but because in worlds in which SGCA was built (and wasn't aligned), ...
A smart enough LLM might be deadly simply if you run it for long enough
by Mikhail Samin — 2023-05-05T20:49:31.416Z

TL;DR: I introduce a new potential threat: a smart enough LLM, even if it’s myopic, not fine-tuned with any sort of RL, “non-agentic”, and was prompt-engineered into imitating aligned and helpful scientists who are doing their best to help you with your world-saving research, might still kill you if you run it for long enough. A way this might happen, suggested here, is that if during the text generation loop more agentic and context-aware parts of what a smart LLM is thinking about are able to gain more influence over the further tokens by having some influence over the current one, the distribution of processes which the LLM thinks about might be increasingly focused on powerful agents, until the text converges to being produced by an extremely coherent and goal-oriented AGI that controls the model’s outputs, in a way most useful for subgoals that include killing everyone.

Thanks to Janus, Owen Parsons, Slava Meriton, Jonathan Claybrough, Daniel Filan, Justis, Eric, and others who provided feedback on the ideas and the draft.

Epistemics status: This is a short post that tries to convey my intuitions about the dynamic. I’m not confident in the intuitions. Some researchers told me the dynamic seems plausible or likely; I think it might serve as an example of a

sharp left turn

difficulty, when parts of the system transform into something far more coherent and agentic than what they were initially, and alignment properties no longer hold. I’m unsure how realistic this threat is, but maybe some actual work on it would be helpful. Note that the post might be somewhat confusing and I failed to optimise it for being easy to read; the title points in a direction I want to convey.

Imagine you understand you’re in a world that an LLM simulates to predict the next token. You’re in a room where the current part of the story takes place, and there’s a microphone in the room that will, in a couple of seconds, record words spoken nearby. Then, based on the loudness of the words, one will be chosen, and a new distribution of worlds will be generated from the text with the chosen word added. Something resembling you might exist in some of these worlds. After several iterations, someone in the real world will look at the selected text. Take a moment to think: what do you do, if you’re smart and goal-oriented?[1]

The main claim

If the LLM is smart enough to continue a text ab...
What Jason has been reading, May 2023: “Protopia,” complex systems, Daedalus vs. Icarus, and more
by jasoncrawford — 2023-05-05T19:54:16.377Z
Link: https://rootsofprogress.org/reading-2023-05

This is a monthly feature. As usual, I’ve omitted recent blog posts and such, which you can find in my links digests.

John Gall, The Systems Bible (2012), aka Systemantics, 3rd ed. A concise, pithy collection of wisdom about “systems”, mostly human organizations, projects, and programs. A classic, and recommended, although I found it a mixed bag. There is much wisdom in here, but also a lot of cynicism and little to no epistemic rigor: less like a serious writer trying to convince you of something, and more like a crotchety old man lecturing to you from his armchair. He throws out examples dripping with snark, but they felt under-analyzed to me. At one point he casually dismisses basically all of psychiatry. But if you can get past all of that, or if you just go into it knowing what to expect, there are a lot of deep lessons, e.g:

A complex system that works is invariably found to have evolved from a simple system that worked. … A complex system designed from scratch never works and cannot be made to work. You have to start over, beginning with a working simple system.

or:

Any large system is going to be operating most of the time in failure mode. What the System is supposed to be doing when everything is working well is really beside the point, because that happy state is rarely achieved in real life. The truly pertinent question is: How does it work when its components aren’t working well? How does it fail? How well does it work in Failure Mode?

For a shorter and more serious treatment of some of the same topics, see “How Complex Systems Fail” (which I covered in a previous reading list).

I’m still perusing Matt Ridley’s How Innovation Works (2020). One story I enjoyed was, at long last, an answer to the question of why we waited so long for the wheeled suitcase, invented by Bernard Sadow in 1970. People love to bring up this example in the context of “ideas behind their time” (although in my opinion it’s not a very strong example because it’s a relatively minor improvement). Anyway, it turns out that the need for wheels on suitcases was far from obvious:

… when Sadow took his crude prototype to retailers, one by one they turned him down. The objections were many and varied. Why add the weight of wheels to a suitcase when you could put it on a baggage trolley or hand it to a porter? Why add to the cost?

Also, as often (always?) happens in the history of invention,...
CHAT Diplomacy: LLMs and National Security
by SebastianG  — 2023-05-05T19:45:27.507Z

"The view, expressed by almost all competent atomic scientists, that there was no "secret" about how to build an atomic bomb was thus not only rejected by influential people in the U.S. political establishment, but was regarded as a treasonous plot."

Robert Oppenheimer A Life at the Center, Ray Monk.

[This essay addresses the probability and existential risk of AI through the lens of national security, which the author believes is the most impactful way to address the issue. Thus the author restricts application of the argument to specific near term versions of Processes for Automating Scientific and Technological Advancement (PASTAs) and human-level AI.]

--

Are Advances in LLMs a National Security Risk?

“This is the number one thing keeping me up at night... reckless, rapid development. The pace is frightening... It is teaching itself more than we programmed it for."

Kiersten Todt, Chief of Staff, CISA [1]

National security apparati around the world should see advances in LLMs as a risk that requires joint-diplomatic efforts, even with current adversaries. This paper explores why this may be the case.

The key steps to the argument are thus:

1. The cost of malicious and autonomous attacks is falling.

2. If the cost of defense is not falling at the same rate, then the current balance of forces between cybersecurity and cyberattack will favor attack, creating a cascade of vulnerabilities across critical systems.

3. The proliferation of these generative AI models (at least across governments).

4. Thus security and information sharing between public and private sectors will be essential for ensuring best practices in security and defense.

5. But the number of vulnerabilities is also increasing. Thus the potential for explosive escalation and/or destabilization of regimes will be great. Non-state actors will increasingly be able to operate in what were previously nation-state level activities.

6. Thus I conclude that capabilities monitoring and both public-private and joint-diplomatic efforts are likely required to protect citizen interests and prevent human suffering.

The Cost of Attacks is Falling

The cost of sophisticated attacks has been falling for some time. Cybersecurity insurance costs indicate that defense is losing to offense [2]. The market rates for insurance indicate that the liability is increasing faster than defensive protocols can b...
Linkpost for Accursed Farms Discussion / debate with AI expert Eliezer Yudkowsky
by gilch — 2023-05-05T18:20:20.004Z
Link: https://www.youtube.com/watch?v=hxsAuxswOvM

Eliezer has been on a podcast tour of late. This appears to be the latest one on YouTube.

I feel like this one didn't go so well. It's also over two hours, so maybe don't watch this one if you have better things to do, but for those that did, I thought I'd make a space for discussion.

Most of it was Eliezer trying to do some Socratic questioning to try to get to the bottom of the host's intuition that AI isn't "really" intelligent, or just imitates it in fragments, and therefore is not dangerous. I'm not sure I understood. It didn't seem to be expressed very well.

Eliezer's goal in doing the interview seems to have been to figure out how to talk to a different audience. People in the know ask very different questions from those who are just learning about this.
Regulate or Compete? The China Factor in U.S. AI Policy (NAIR #2)
by charles_m — 2023-05-05T17:43:42.417Z
Link: https://navigatingairisks.substack.com/p/regulate-or-compete-china-factor-us-ai-policy

This post was published as part of the 2nd edition of Navigating AI Risks, a newsletter about the governance of transformative AI risks written by @simeon_c, @Henry Papadatos, and me. Feel free to submit your feedback on the newsletter here.

Summary

The supposed trade-off between regulation and technology leadership is widely put forward as an argument for avoiding regulation. But currently, preservation by the US of its AI leadership over China need not come at the cost of effective AI governance.

In the context of US-China strategic competition, two key variables define policymakers' risk-benefit analysis regarding AI regulation:

the desirability of AI regulation and its impact on innovation capacity
whether the United States is on course to maintain (or increase) its AI lead over China

Here are my observations on the first variable (desirability of AI regulation):

Regulation can promote and shape innovation
Increasingly powerful systems are rendering existing laws insufficient, and regulation helps avoid large-scale accidents
Regulation can foster better regulation
Regulation helps facilitate adoption

Here are my observations on the second variable (likelihood that the US maintains its AI lead over China):

China has ambitious goals and is putting in the means to achieve them
China currently remains behind the US
China's autocratic advantage is overestimated
China doesn’t prioritize AI leadership at all costs
Innovation is insufficient by itself to get ahead
The US has a well-developed network of alliances it can use to its benefit

I conclude that (i) the United States' AI lead over China and (ii) the risks posed by AI seem large enough to set up guardrails, even if it stifles innovation. The United States doesn’t have to choose between maintaining AI leadership over China and hardwiring both safety procedures and democratic values into its AI governance ecosystem. It should do both.

Introduction

The modern story of tech regulation in the United States is a story about China. Worries that regulation may damage the US's technology leadership vis-à-vis China have often been cited as a reason not to regulate. This concern played an important role in

debates

on privacy rules[1], breaking up Big Tech companies[2], and now AI[3].

This trade-off between regulation and technological preeminence seems obvious. But the situation is more complicated. The preservation of AI le...
Kingfisher Live CD Process
by jefftk — 2023-05-05T17:00:04.651Z

Our live album is

out
today! We'll have physical CDs around the end of May, and it
should be on streaming services at some point. If you backed the
Kickstarter and haven't heard from us about your rewards, let me know!

Some background on the album if you're interested:

In late January I wrote to Cecilia to ask what she'd
think of making a live album. I'd been making good quality
multi-track recordings at most of our gigs, and we had a lot to work
with. I sent her the first email at 5:38pm and by 8:06pm we had not
only agreed we wanted to do this but also agreed we both wanted Dana Billings for editing and
mixing. I wrote to Dana, he liked the idea, and we told him we'd get
back to him once we'd listened back over our tracks to figure out which
ones were our favorites.

This was a long and painful process: we had recordings from 146 sets
across 11 gigs, for a total of ~20hr of music. I went through and
made scratch mixes of everything, and made a giant spreadsheet for
collecting ratings. The initial idea was that we'd both rate each
track, but since I had more time (I could listen while watching kids
[1]) for many of them I did an initial rating and Cecilia only rated
the ones I'd marked as potentially good enough.

By early March we were pretty far along and I wrote something
up to gauge interest and ask how people usually funded albums
these days. Several people recommended Kickstarter, so we decided
we'd probably do that. A few days later we had agreement on three
tracks, Griffin
Road, Trip
to Moscow, and Salt
River, and we sent Dana what he needed to get started with mixing.
This included both the raw multitrack recordings, and edit notes. For
example, on Griffin Road the initial edit notes were:

cut the second time through since the fiddle delay didn't work

could cut the fourth time through for length

the timing is off going into the eighth time through; could cut this
time through or fix?

the timing is off eight beats into what I think is the beginning
of the 13th time through (~6:45) -- fix or cut?

probably find 2-4 other times through to cut for length and interest?

A week later Dana got back to us with the first mix, for Griffin Road,
and we were really happy. Even having listened to the scratch mix
repeatedly I couldn't tell where the edits were: it just sounded
good. Over the next few days he got bac...
What can we learn from Bayes about reasoning?
by jasoncrawford — 2023-05-05T15:52:35.980Z

What are the qualitative lessons we can learn about logic and reasoning from Bayesian epistemology, that is, from taking Bayes' rule as a mathematical model for thought (even if it is considered a simplified formalism that we often can't implement?)

I've seen at least a few of these from @Eliezer Yudkowsky, but I think they're scattered across many essays.

Some things I consider to be examples of what I'm gesturing at here:

Aumann's Agreement Theorem
Conservation of expected evidence

Thanks!
Why not use active SETI to prevent AI Doom?
by RomanS — 2023-05-05T14:41:40.661Z

Let's assume that Eliezer is right: soon we'll have an AGI that is very likely to kill us all. (personally, I think Eliezer is right).

There are several ways to reduce the risk, in particular: speeding up alignment research and slowing down capabilities research, by various means.

One underexplored way to reduce the risk is active SETI (also known as METI).

The idea is as follows:

Send powerful radio signals into space: "guys, soon we'll be destroyed by a hostile AGI. Help us!" (e.g. using a language constructed for the task, like Lincos)
If a hostile alien civilization notices us, we're going to die. But if we're going to die from the AGI anyway, who cares?
If a benevolent alien civilization notices us, it could arrive in time to save us.

The main advantage of the method is that it can be implemented by a small group of people within a few months, without governments and without billions of dollars. Judging by the running costs of the Arecibo Observatory, one theoretically can rent it for a year for only $8 million. Sending only a few hundred space messages could be even cheaper.

Obviously, the method relies on the existence of an advanced alien civilization within a few light years from the Earth. The existence seems to be unlikely, but who knows.

Is it worth trying?
Investigating Emergent Goal-Like Behavior in Large Language Models using Experimental Economics
by phelps-sg — 2023-05-05T11:15:13.390Z

Epistemic status: My Best Guess (some preliminary empirical evidence- more experiments and critique required)

This post introduces an early-draft working paper investigating the propensity of LLMs to cooperate and reciprocate in the Prisoner's Dilemma: https://sphelps.net/papers/llm-cooperation. I plan on submitting to arXiv after incorporating feedback from the community. An edited version of the paper's introduction is given below.

TLDR; there has been some discussion on applying ideas from game theory to bring the incentives of AI agents into alignment with human goals, but as far as I know there has been little empirical investigation of how actual AIs behave in cooperative tasks when faced with e.g. conditional altruism (though happy to be corrected). We decided to investigate this empirically by running experiments. Preliminary investigation with GPT-3.5 indicates that LLMs can operationalise natural language descriptions of altruism and selfishness into appropriate behaviour in the iterated PD to some extent, but exhibit limitations in adapting their behavior based on conditioned reciprocity.

Motivation and background

Many scenarios discussed in the AI alignment debate focus on competitive zero-sum interactions. For example, a common analogy is competition for resources between different species occupying a particular niche; for example, [Tegmark, 2023] argues

We humans drove the West African Black Rhino extinct not because we were rhino-haters, but because we were smarter than them and had different goals for how to use their habitats and horns. In the same way, superintelligence with almost any open-ended goal would want to preserve itself and amass resources to accomplish that goal better.

In an AI safety context, the intuition behind such arguments is that AI systems have been to shown to outsmart humans in zero-sum games such as Chess and Go, and therefore if AI systems find themselves in situations in which they are competing with humans, the AI “species” will clearly out-compete inferior humans.

However, many interactions in both natural and artificial settings are characterized by non-zero-sum payoff structures [Phelps and Russell, 2015]. A famous example that was used to analyse existential risk of nuclear conflict during the cold war is the Prisoner’s Dilemma [Axelrod, 1997]. In an ecological context a related n...